Nov 03 01:14:50.116977 ip-10-0-138-19 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 01:14:50.116991 ip-10-0-138-19 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 01:14:50.117000 ip-10-0-138-19 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 01:14:50.117309 ip-10-0-138-19 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 01:15:00.143308 ip-10-0-138-19 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 01:15:00.143324 ip-10-0-138-19 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 786a1ea8fd074a528c757cef5123c06b -- Nov 03 01:17:00.145305 ip-10-0-138-19 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:17:00.563408 ip-10-0-138-19 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:00.563408 ip-10-0-138-19 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:17:00.563408 ip-10-0-138-19 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:00.563408 ip-10-0-138-19 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:00.567109 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567021 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.567109 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567108 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.567109 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567111 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567118 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567123 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567127 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567130 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567133 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567136 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567139 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567142 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567145 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567148 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567151 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567154 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567156 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567159 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567162 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567165 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567167 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567170 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.567201 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567179 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567182 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567184 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567187 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567190 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567192 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567195 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567197 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567200 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567203 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567206 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567208 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567211 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567214 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567217 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567219 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567222 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567224 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567227 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567229 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.567640 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567232 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567234 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567237 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567240 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567243 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567245 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567249 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567251 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567254 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567257 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567259 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567261 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567264 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567266 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567270 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567273 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567279 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567282 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.568124 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567285 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567288 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567290 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567293 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567296 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567298 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567301 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567303 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567306 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567309 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567311 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567314 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567316 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567319 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567321 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567323 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567326 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567329 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567331 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.568547 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567334 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567337 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567340 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567343 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567346 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567348 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567351 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567354 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567356 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567361 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567363 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567366 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567369 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567371 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567374 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567377 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567379 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567382 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567384 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567387 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.569027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567389 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567392 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567395 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567397 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567400 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567402 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567405 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.567407 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.568986 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.568994 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.568997 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569000 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569003 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569019 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569022 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569026 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569028 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569031 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569034 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569037 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.569516 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569040 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569044 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569047 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569049 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569052 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569055 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569057 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569060 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569062 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569065 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569067 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569070 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569072 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569075 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569078 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569083 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569088 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569091 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569094 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.570048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569097 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569100 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569104 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569107 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569109 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569112 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569115 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569117 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569120 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569123 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569126 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569129 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569131 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569134 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569137 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569139 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569142 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569145 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569147 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569150 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.570518 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569152 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569155 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569158 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569160 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569163 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569165 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569168 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569172 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569175 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569178 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569180 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569184 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569187 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569189 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569192 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569194 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569197 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569199 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569202 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569204 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.570986 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569207 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569209 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569213 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569215 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569218 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569220 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569223 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569225 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569228 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569231 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569233 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569235 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569238 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569240 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569243 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569247 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569251 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569254 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569257 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.571508 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569261 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569264 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569268 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569271 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569273 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569276 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569278 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569280 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569283 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569285 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569288 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569293 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569296 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569298 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569301 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569782 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569788 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569792 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569795 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.571983 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569798 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569801 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569804 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569806 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569809 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569811 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569814 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569817 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569820 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569822 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569825 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569828 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569831 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569834 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569837 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569839 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569842 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569845 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569847 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.572505 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569850 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569852 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569855 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569858 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569860 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569863 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569865 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569868 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569870 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569873 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569876 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569879 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569882 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569884 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569887 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569889 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569893 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569896 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569898 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569901 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.573045 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569904 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569906 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569909 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569911 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569914 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569918 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569921 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569923 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569926 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569928 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569931 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569933 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569936 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569939 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569941 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569944 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569946 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569949 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569951 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569953 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.573581 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569956 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569958 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569964 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569967 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569969 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569973 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569975 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569978 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569981 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569984 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569986 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569989 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569992 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569994 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569997 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.569999 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570002 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570005 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570022 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570025 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.574140 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570027 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570030 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570033 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570036 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570039 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570041 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570044 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570047 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570049 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570052 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570055 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570057 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570060 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570063 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570067 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570070 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570073 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570076 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570079 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.574669 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570082 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570084 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570087 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570090 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570176 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570183 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570189 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570194 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570198 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570201 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570206 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570210 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570213 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570216 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570220 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570224 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570227 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570230 2573 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570233 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570236 2573 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570239 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570242 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570247 2573 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:17:00.575202 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570250 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570253 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570256 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570259 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570265 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570268 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570272 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570275 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570278 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570281 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570284 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570288 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570291 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570295 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570298 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570301 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570304 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570307 2573 flags.go:64] FLAG: --enable-server="true" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570310 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570315 2573 flags.go:64] FLAG: --event-burst="100" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570318 2573 flags.go:64] FLAG: --event-qps="50" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570321 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570324 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570327 2573 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:17:00.575792 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570331 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570334 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570338 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570341 2573 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570344 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570347 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570350 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570353 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570356 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570359 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570362 2573 flags.go:64] FLAG: --feature-gates="" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570369 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570373 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570377 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570380 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570384 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570387 2573 flags.go:64] FLAG: --help="false" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570390 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570393 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570397 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570399 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570403 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570407 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:17:00.576508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570410 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570413 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570416 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570419 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570422 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570425 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570428 2573 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570431 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570434 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570437 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570440 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570443 2573 flags.go:64] FLAG: --lock-file="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570446 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570449 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570452 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570457 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570461 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570464 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570467 2573 flags.go:64] FLAG: --logging-format="text" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570470 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570473 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570477 2573 flags.go:64] FLAG: --manifest-url="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570480 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570485 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570488 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:17:00.577086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570493 2573 flags.go:64] FLAG: --max-pods="110" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570496 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570499 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570502 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570505 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570508 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570511 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570514 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570522 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570525 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570528 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570531 2573 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570534 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570537 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570540 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570543 2573 flags.go:64] FLAG: --port="10250" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570547 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570550 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00701afe6b45be27b" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570553 2573 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570556 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570559 2573 flags.go:64] FLAG: --register-node="true" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570562 2573 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570566 2573 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570569 2573 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570572 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570575 2573 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:17:00.577787 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570578 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570582 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570587 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570590 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570593 2573 flags.go:64] FLAG: --runonce="false" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570596 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570599 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570603 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570606 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570611 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570614 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570617 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570620 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570623 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570626 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570629 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570632 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570635 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570638 2573 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570641 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570646 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570649 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570652 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570656 2573 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570659 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:17:00.578432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570662 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570665 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570668 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570671 2573 flags.go:64] FLAG: --v="2" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570675 2573 flags.go:64] FLAG: --version="false" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570679 2573 flags.go:64] FLAG: --vmodule="" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570683 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.570687 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570821 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570826 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570829 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570831 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570834 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570837 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570840 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570843 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570847 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570850 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570853 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570855 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570858 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570861 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.579027 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570863 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570866 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570868 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570872 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570874 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570877 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570879 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570882 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570885 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570887 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570890 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570892 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570895 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570898 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570900 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570904 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570907 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570911 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570913 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570917 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.579593 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570920 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570922 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570925 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570928 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570930 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570933 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570937 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570940 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570943 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570945 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570948 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570951 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570953 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570956 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570958 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570961 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570964 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570966 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570969 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.580083 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570971 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570974 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570976 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570979 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570982 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570984 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570987 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570990 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570992 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570995 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.570998 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571000 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571004 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571018 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571021 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571023 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571026 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571029 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571033 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.580536 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571038 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571043 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571046 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571049 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571052 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571054 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571057 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571060 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571062 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571065 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571067 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571070 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571073 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571075 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571078 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571080 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571089 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571092 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571095 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571097 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.581084 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571100 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571103 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571105 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571108 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571111 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571114 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571117 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571119 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571122 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571124 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571127 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571129 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571134 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571136 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.571164 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571206 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.581564 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571210 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571213 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571216 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571219 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571221 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571224 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571227 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571230 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571232 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571235 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571238 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571240 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571243 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571245 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571248 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571251 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571254 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571257 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571259 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571262 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.581948 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571265 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571267 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571270 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571273 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571276 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571278 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571281 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571283 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571289 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571293 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571297 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571302 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571305 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571308 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571310 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571313 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571315 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571318 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571321 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571323 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.582454 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571326 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571328 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571331 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571334 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571336 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571339 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571341 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571344 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571346 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571349 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571351 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571354 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571356 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571359 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571362 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571364 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571367 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571369 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571372 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.582934 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571374 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571382 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571384 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571387 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571389 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571392 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571395 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571397 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571400 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571402 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571405 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571409 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571412 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571415 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571417 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571420 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571422 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571425 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571427 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.583400 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571430 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571432 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571435 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571437 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571440 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571442 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571445 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571448 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571452 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571454 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571457 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571459 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571462 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571465 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571468 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571471 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571473 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571476 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571479 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571482 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.583859 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571485 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571488 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571491 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571494 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571496 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571499 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.571501 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.578920 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.578937 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.578954 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.578959 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579030 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579035 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579038 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579041 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579044 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579047 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579050 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579053 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.584339 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579058 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579061 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579064 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579067 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579070 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579073 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579075 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579078 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579081 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579091 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579094 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579097 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579101 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579104 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579107 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579109 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579113 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579116 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579118 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.584795 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579121 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579124 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579127 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579129 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579132 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579135 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579137 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579140 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579142 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579145 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579147 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579150 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579152 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579155 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579158 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579162 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579167 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579171 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.585273 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579175 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579178 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579181 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579183 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579186 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579188 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579191 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579193 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579196 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579199 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579202 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579205 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579207 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579210 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579212 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579215 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579218 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579220 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579223 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579225 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.585699 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579228 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579230 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579233 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579235 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579238 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579241 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579244 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579246 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579249 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579251 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579253 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579256 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579258 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579261 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579264 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579266 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579269 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579271 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579274 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579276 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.586179 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579279 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579282 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579285 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579288 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579290 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579293 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579296 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579299 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579301 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579304 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579306 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579308 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579311 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579313 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579316 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579319 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579321 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579323 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579326 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.586733 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579328 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.579355 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579494 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579499 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579502 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579505 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579507 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579510 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579513 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579515 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579518 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579521 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579523 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579526 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579528 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579535 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.587228 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579537 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579540 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579543 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579545 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579549 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579553 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579556 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579559 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579562 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579564 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579567 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579569 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579572 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579574 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579577 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579579 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579582 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579584 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579587 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.587770 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579590 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579592 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579595 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579598 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579601 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579603 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579606 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579609 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579612 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579614 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579617 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579619 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579622 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579625 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579628 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579630 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579633 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579636 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579638 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579641 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.588404 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579643 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579646 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579648 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579650 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579653 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579655 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579658 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579660 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579663 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579665 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579668 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579670 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579673 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579675 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579678 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579683 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579685 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579688 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579690 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.588909 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579694 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579697 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579700 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579703 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579706 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579708 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579711 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579714 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579717 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579719 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579722 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579724 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579727 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579729 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579732 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579734 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579737 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579739 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579742 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.589413 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579744 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579747 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579749 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579752 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579754 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579757 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579759 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579762 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579764 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579769 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579771 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579774 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579777 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579779 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579781 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.579804 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:00.589907 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579834 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579838 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579841 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579844 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579847 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579850 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579852 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579855 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579857 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579860 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579862 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579865 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579867 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579870 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579873 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579875 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579878 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579880 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579882 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:00.590448 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579885 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579888 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579890 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579893 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579895 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579900 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579902 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579906 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579908 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579911 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579914 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579917 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579919 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579922 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579924 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579927 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579930 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579933 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579935 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579938 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:00.590896 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579941 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579944 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579947 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579949 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579952 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579954 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579958 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579962 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579965 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579968 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579971 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579973 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579976 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579979 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579981 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579983 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579986 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579990 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579992 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:00.591399 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.579997 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580000 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580003 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580018 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580023 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580027 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580031 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580035 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580039 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580042 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580044 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580047 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580050 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580053 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580056 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580059 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580061 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580064 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580066 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:00.591864 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580069 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580072 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580075 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580077 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580080 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580083 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580085 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580088 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580091 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580094 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580097 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580101 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580103 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580107 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580110 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580113 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580115 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580118 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580120 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580123 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:00.592325 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580126 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580128 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580133 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580136 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580139 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580142 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580145 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580148 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:00.580150 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.580221 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.583738 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.584738 2573 server.go:1014] "Starting client certificate rotation" Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.586578 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.586805 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:17:00.592798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.589912 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:00.595806 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.595791 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:00.605738 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.605719 2573 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:17:00.608828 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.608814 2573 log.go:25] "Validated CRI v1 image API" Nov 03 01:17:00.611479 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.611455 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:17:00.615765 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.615740 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86498aeb-34e8-431d-813e-47863e9e50a1:/dev/nvme0n1p4 f787a52f-e48a-43b3-a9dd-a492aae411f6:/dev/nvme0n1p3] Nov 03 01:17:00.615811 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.615765 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 01:17:00.621140 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.621028 2573 manager.go:217] Machine: {Timestamp:2026-11-03 01:17:00.619504674 +0000 UTC m=+0.348267251 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098528 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e3a88865684c08a72dfbb2bca4d51 SystemUUID:ec2e3a88-8656-84c0-8a72-dfbb2bca4d51 BootID:786a1ea8-fd07-4a52-8c75-7cef5123c06b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e2:da:d9:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e2:da:d9:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:ae:d3:a2:7e:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:17:00.621823 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.621813 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:17:00.621942 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.621930 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:17:00.625484 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.625461 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:17:00.625631 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.625486 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-19.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:17:00.626306 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.626295 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:17:00.626343 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.626308 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:17:00.626343 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.626336 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:00.626395 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.626352 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:00.627645 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.627634 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:17:00.632308 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.632296 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:17:00.632430 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.632421 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:17:00.633340 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633329 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:17:00.633370 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633346 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:17:00.633370 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633357 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:17:00.633370 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633366 2573 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:17:00.633480 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633375 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:17:00.633741 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.633728 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:00.636888 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.636867 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:17:00.644789 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.644771 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:00.645776 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.645762 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:17:00.645858 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.645802 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:17:00.645999 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.645987 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:17:00.646074 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646025 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:17:00.646074 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646036 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:17:00.646074 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646045 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:17:00.646074 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646055 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:17:00.646074 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646064 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:17:00.646691 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646681 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:17:00.646746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646694 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:17:00.646746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646706 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:17:00.646746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646716 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:17:00.646746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646729 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:17:00.646746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.646743 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:17:00.647532 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.647520 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:17:00.647596 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.647536 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:17:00.651411 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.651395 2573 server.go:1263] "Started kubelet" Nov 03 01:17:00.651859 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.651475 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:17:00.651859 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.651485 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:17:00.651859 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.651543 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:17:00.652181 ip-10-0-138-19 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:17:00.653163 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.653147 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:17:00.653526 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.653508 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:17:00.661357 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.661336 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:17:00.662088 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.662065 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 01:17:00.662639 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.662625 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:17:00.662755 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.662728 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663468 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663505 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663517 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663621 2573 factory.go:55] Registering systemd factory Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663635 2573 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663642 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663652 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663979 2573 factory.go:153] Registering CRI-O factory Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.663995 2573 factory.go:223] Registration of the crio container factory successfully Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.664061 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.664076 2573 factory.go:103] Registering Raw factory Nov 03 01:17:00.664557 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.664086 2573 manager.go:1199] Started watching for new ooms in manager Nov 03 01:17:00.665318 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.665304 2573 manager.go:319] Starting recovery of all containers Nov 03 01:17:00.676118 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.676097 2573 manager.go:324] Recovery completed Nov 03 01:17:00.681087 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.681073 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.682773 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.682758 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.682845 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.682786 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.682845 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.682796 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.683282 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683269 2573 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:17:00.683325 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683281 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:17:00.683325 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683301 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:17:00.683325 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683311 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.683959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683947 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.684003 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683972 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.684003 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.683982 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.685562 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.685549 2573 policy_none.go:50] "Start" Nov 03 01:17:00.685611 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.685565 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:17:00.685611 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.685578 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:17:00.687606 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.687594 2573 policy_none.go:44] "Start" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.728721 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.744228 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.745489 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.745524 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.745551 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.745593 2573 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748181 2573 manager.go:341] "Starting Device Plugin manager" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.748223 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748236 2573 server.go:87] "Starting device plugin registration server" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748507 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:17:00.748546 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748520 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:17:00.749387 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748637 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:17:00.749387 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748724 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:17:00.749387 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.748731 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:17:00.749387 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.749214 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:17:00.749387 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.749274 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:00.789221 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.789185 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 01:17:00.789221 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.789197 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 01:17:00.790351 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.789210 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.us-west-2.compute.internal.18e4641b72b08d58 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:00.651367768 +0000 UTC m=+0.380130347,LastTimestamp:2026-11-03 01:17:00.651367768 +0000 UTC m=+0.380130347,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.us-west-2.compute.internal,}" Nov 03 01:17:00.846964 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.846896 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal"] Nov 03 01:17:00.847102 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.846974 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.847964 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.847946 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.848083 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.847976 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.848083 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.847986 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.849032 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.849004 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.849693 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.849669 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.849769 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.849703 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.849769 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.849719 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.849769 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.849742 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.850400 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.850384 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.850536 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.850522 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.850595 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.850583 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.851003 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.850988 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.851095 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.851033 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.851095 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.851044 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.851095 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.850990 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.851192 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.851124 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.851192 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.851135 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.853191 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.853176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.853251 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.853202 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:00.854520 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.854501 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:00.854614 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.854529 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:00.854614 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.854539 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:00.865240 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.865220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.865329 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.865244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.865329 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.865259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.878553 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.878534 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:17:00.878603 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.878582 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjqrm" Nov 03 01:17:00.879913 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.879897 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.882162 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:00.882148 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.939193 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.939163 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:00.965705 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.965705 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.965875 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.965875 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965757 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.965875 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965768 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.965875 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.965778 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:00.983618 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:00.983590 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:01.030032 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.029987 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:01.031035 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.031017 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:01.031154 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:01.031042 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-19.us-west-2.compute.internal\": node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:01.031606 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.031588 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:01.062974 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.062948 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:01.163285 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.163204 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:01.163446 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.163325 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:01.184648 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.184625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:01.187790 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.187772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:01.254308 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:01.254147 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:02.904399 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.904311 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 01:17:02.904923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.904471 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:17:02.904923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.904496 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:17:02.904923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.904497 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 01:17:02.904923 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:02.904541 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aece31dc849e6402abdece8830c7a683-e1048f7a7a1c1f18.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-19.us-west-2.compute.internal.18e4641b748fc0d9\": read tcp 10.0.138.19:33028->3.150.166.25:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-19.us-west-2.compute.internal.18e4641b748fc0d9 default 4825 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:00 +0000 UTC,LastTimestamp:2026-11-03 01:17:00.849687915 +0000 UTC m=+0.578450493,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.us-west-2.compute.internal,}" Nov 03 01:17:02.952471 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.952439 2573 apiserver.go:52] "Watching apiserver" Nov 03 01:17:02.980738 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:02.980711 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 01:17:03.203906 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.203804 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:02 +0000 UTC" deadline="2028-06-04 04:44:37.327735974 +0000 UTC" Nov 03 01:17:03.203906 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.203850 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13899h27m34.123891247s" Nov 03 01:17:03.218733 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.218704 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:17:03.219036 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.219019 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rv9vt","openshift-image-registry/node-ca-jkcgz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-qk55d","kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6g9zq","openshift-multus/multus-vfccq","openshift-multus/network-metrics-daemon-x9lj4","kube-system/global-pull-secret-syncer-xqklw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6","openshift-network-operator/iptables-alerter-9wq78","openshift-ovn-kubernetes/ovnkube-node-8k5j6","kube-system/konnectivity-agent-tjn7g","openshift-cluster-node-tuning-operator/tuned-d64cz"] Nov 03 01:17:03.221495 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.221479 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.223568 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.223529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.225629 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.225614 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:03.225694 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.225675 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:03.227770 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.227753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.229829 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.229814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.231917 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.231903 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.231968 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.231950 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:03.234100 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.234079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.234182 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.234148 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:03.236329 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.236317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.238425 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.238412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.240783 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.240768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.243526 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.243506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.245769 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.245752 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.274697 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.274672 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.274805 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.274678 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.274805 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.274766 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\"" Nov 03 01:17:03.275774 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.275753 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.275852 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.275785 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\"" Nov 03 01:17:03.276026 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.276007 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:17:03.276026 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.276022 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.280313 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280295 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:17:03.280387 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280329 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.280665 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280645 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:17:03.280754 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280649 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\"" Nov 03 01:17:03.280812 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280800 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.280847 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.280824 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:17:03.282107 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.282094 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:17:03.282319 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.282307 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\"" Nov 03 01:17:03.282883 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.282870 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:17:03.288648 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.288630 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.288707 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.288642 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:17:03.288707 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.288636 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.288707 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.288697 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\"" Nov 03 01:17:03.290398 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.290482 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffq6\" (UniqueName: \"kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.290482 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.290482 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t67t\" (UniqueName: \"kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.290658 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5j5g\" (UniqueName: \"kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.290658 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhc7\" (UniqueName: \"kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.290658 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290658 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290636 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.290658 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbb8\" (UniqueName: \"kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290790 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kbl\" (UniqueName: \"kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.290904 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290955 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.290989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6j4\" (UniqueName: \"kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.291287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291268 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291324 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291472 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291585 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.291999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld79v\" (UniqueName: \"kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292053 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.292711 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65hc\" (UniqueName: \"kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292398 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.293164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.292478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.321508 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.321490 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\"" Nov 03 01:17:03.321594 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.321516 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\"" Nov 03 01:17:03.321650 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.321637 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\"" Nov 03 01:17:03.321860 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.321848 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:17:03.324849 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.324831 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\"" Nov 03 01:17:03.324849 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.324834 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:17:03.325006 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.324991 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.325060 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.325051 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.325109 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.325073 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.325109 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.325080 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:17:03.326358 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.326344 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:03.326412 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.326377 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:17:03.330765 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.330751 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.332780 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.332766 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:17:03.332780 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.332774 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:17:03.332867 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.332786 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:03.332867 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.332799 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:17:03.338870 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.338855 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 01:17:03.338945 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.338889 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xggl7" Nov 03 01:17:03.393698 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393742 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6j4\" (UniqueName: \"kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393801 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.393836 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393822 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393843 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394005 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394031 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394107 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394139 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394141 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.394168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394138 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.393914 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394205 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394227 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394289 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394329 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394341 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.394963 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394374 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394420 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394434 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394534 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394536 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.395468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394640 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394667 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394737 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394835 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394895 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.396257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394936 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld79v\" (UniqueName: \"kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.394999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395060 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395206 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395417 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.397082 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65hc\" (UniqueName: \"kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395676 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395726 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395785 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.395805 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395856 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.395868 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:03.895844619 +0000 UTC m=+2.306058221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.397923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395909 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395914 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffq6\" (UniqueName: \"kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t67t\" (UniqueName: \"kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.395991 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5j5g\" (UniqueName: \"kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhc7\" (UniqueName: \"kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbb8\" (UniqueName: \"kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.398713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396304 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396324 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kbl\" (UniqueName: \"kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396312 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396350 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396402 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396438 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.396544 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:03.399147 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.396601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:03.896587181 +0000 UTC m=+2.306800776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.396960 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.397057 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.398044 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.398323 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.398351 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.398875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.399627 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.398897 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.453767 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.453735 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:03.453767 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.453757 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:03.453767 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.453768 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:03.454052 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.453821 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:03.953806294 +0000 UTC m=+2.364019891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:03.455480 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.455446 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6j4\" (UniqueName: \"kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.455955 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.455932 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld79v\" (UniqueName: \"kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.456713 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.456689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffq6\" (UniqueName: \"kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.457440 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.457419 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t67t\" (UniqueName: \"kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.457513 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.457439 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65hc\" (UniqueName: \"kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.457513 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.457486 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5j5g\" (UniqueName: \"kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.457727 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.457710 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbb8\" (UniqueName: \"kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.457999 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.457982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhc7\" (UniqueName: \"kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.458648 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.458633 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kbl\" (UniqueName: \"kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.554663 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.554629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:03.578936 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.578908 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:03.607980 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.607950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:03.615541 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.615522 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vfccq" Nov 03 01:17:03.619661 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.619640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:03.647330 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.647307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:03.677833 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.677544 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:03.679175 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.679155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:03.683259 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.683239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:03.785363 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:03.785315 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5781414dea338e9e0fd077df61848e.slice/crio-4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790 WatchSource:0}: Error finding container 4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790: Status 404 returned error can't find the container with id 4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790 Nov 03 01:17:03.785990 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:03.785968 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ac7307a30e6ec216df9c122c45451f.slice/crio-f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60 WatchSource:0}: Error finding container f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60: Status 404 returned error can't find the container with id f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60 Nov 03 01:17:03.790652 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.789798 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:17:03.900051 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.900017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:03.900218 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.900072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:03.900218 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.900151 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:03.900218 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.900152 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:03.900218 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.900197 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:04.900185767 +0000 UTC m=+3.310399350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:03.900218 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:03.900209 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:04.900203344 +0000 UTC m=+3.310416926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:03.996052 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:03.996026 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:04.001207 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.001186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:04.001357 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.001342 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:04.001396 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.001361 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:04.001396 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.001372 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:04.001456 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.001419 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:05.001407534 +0000 UTC m=+3.411621152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:04.068755 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.068663 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"b5ac7307a30e6ec216df9c122c45451f","Type":"ContainerStarted","Data":"f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60"} Nov 03 01:17:04.069701 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.069671 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerStarted","Data":"4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790"} Nov 03 01:17:04.328927 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.328846 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:04.342045 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.341998 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:03 +0000 UTC" deadline="2028-03-31 03:49:53.719570709 +0000 UTC" Nov 03 01:17:04.342045 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.342041 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12338h32m49.377531504s" Nov 03 01:17:04.502956 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.502923 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0657ec93_7ca2_415d_8ce6_d7a6be73d8fa.slice/crio-3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397 WatchSource:0}: Error finding container 3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397: Status 404 returned error can't find the container with id 3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397 Nov 03 01:17:04.517746 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.517726 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:04.558021 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.557989 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764c5ea1_9edc_4a86_8bec_b4001b6ad8c8.slice/crio-53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c WatchSource:0}: Error finding container 53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c: Status 404 returned error can't find the container with id 53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c Nov 03 01:17:04.565740 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.565714 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91186c34_ad26_42d9_a4e4_70f9da7a10db.slice/crio-f52dab486745ec0f30bd924e50a71b665cfe565d98866b7bc55cd28a3c39b641 WatchSource:0}: Error finding container f52dab486745ec0f30bd924e50a71b665cfe565d98866b7bc55cd28a3c39b641: Status 404 returned error can't find the container with id f52dab486745ec0f30bd924e50a71b665cfe565d98866b7bc55cd28a3c39b641 Nov 03 01:17:04.566326 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.566299 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d87eeb_c7ba_4afa_bb30_9a512102f939.slice/crio-30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712 WatchSource:0}: Error finding container 30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712: Status 404 returned error can't find the container with id 30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712 Nov 03 01:17:04.599436 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.599408 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd38c69_7c29_44f2_940f_061783060dcd.slice/crio-ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d WatchSource:0}: Error finding container ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d: Status 404 returned error can't find the container with id ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d Nov 03 01:17:04.600048 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.600025 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954cc14b_f026_4cf9_93cb_b14e3b7ef933.slice/crio-2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409 WatchSource:0}: Error finding container 2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409: Status 404 returned error can't find the container with id 2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409 Nov 03 01:17:04.658953 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.658924 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deae577_9838_4896_82bf_a1b5b5710d3a.slice/crio-96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9 WatchSource:0}: Error finding container 96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9: Status 404 returned error can't find the container with id 96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9 Nov 03 01:17:04.785493 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:04.785463 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac23fba7_e4fe_45bc_8360_9875079da1cd.slice/crio-b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729 WatchSource:0}: Error finding container b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729: Status 404 returned error can't find the container with id b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729 Nov 03 01:17:04.907244 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.907169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:04.907244 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:04.907210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:04.907412 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.907314 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:04.907412 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.907317 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:04.907412 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.907363 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:06.907351476 +0000 UTC m=+5.317565058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:04.907412 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:04.907376 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:06.907369948 +0000 UTC m=+5.317583530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:05.007778 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.007749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:05.008325 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.007901 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:05.008325 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.007921 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:05.008325 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.007930 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:05.008325 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.007980 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:07.007965476 +0000 UTC m=+5.418179057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:05.064993 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.064953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:05.065166 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.065088 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:05.065232 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.065166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:05.065291 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.065233 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:05.065342 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.065291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:05.065415 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:05.065392 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:05.073505 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.073470 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rv9vt" event={"ID":"ac23fba7-e4fe-45bc-8360-9875079da1cd","Type":"ContainerStarted","Data":"b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729"} Nov 03 01:17:05.075630 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.075524 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9"} Nov 03 01:17:05.077429 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.077392 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d64cz" event={"ID":"91186c34-ad26-42d9-a4e4-70f9da7a10db","Type":"ContainerStarted","Data":"f52dab486745ec0f30bd924e50a71b665cfe565d98866b7bc55cd28a3c39b641"} Nov 03 01:17:05.078607 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.078582 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wq78" event={"ID":"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8","Type":"ContainerStarted","Data":"53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c"} Nov 03 01:17:05.080125 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.080104 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkcgz" event={"ID":"43445eef-1fab-461d-a4db-bf26cb747247","Type":"ContainerStarted","Data":"a302280115504712a45c3a7f89cbb427ff50d5da18638af760b92d4b1777e20f"} Nov 03 01:17:05.081485 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.081443 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerStarted","Data":"2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409"} Nov 03 01:17:05.082615 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.082589 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjn7g" event={"ID":"add38c69-7c29-44f2-940f-061783060dcd","Type":"ContainerStarted","Data":"ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d"} Nov 03 01:17:05.084833 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.084808 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vfccq" event={"ID":"31d87eeb-c7ba-4afa-bb30-9a512102f939","Type":"ContainerStarted","Data":"30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712"} Nov 03 01:17:05.086092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.086065 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" event={"ID":"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa","Type":"ContainerStarted","Data":"3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397"} Nov 03 01:17:05.342231 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.342131 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:03 +0000 UTC" deadline="2028-03-28 21:04:57.088991144 +0000 UTC" Nov 03 01:17:05.342231 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:05.342185 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12283h47m51.746815859s" Nov 03 01:17:06.924535 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:06.924493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:06.925112 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:06.924575 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:06.925112 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:06.924720 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:06.925112 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:06.924779 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:10.924762207 +0000 UTC m=+9.334975790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:06.925112 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:06.924844 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:06.925112 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:06.924874 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:10.924863991 +0000 UTC m=+9.335077574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:07.025338 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:07.025279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:07.025508 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.025446 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:07.025508 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.025471 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:07.025508 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.025484 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:07.025723 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.025542 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:11.025524605 +0000 UTC m=+9.435738204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:07.064986 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:07.064945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:07.065157 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.065091 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:07.065221 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:07.065168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:07.065277 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.065242 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:07.065330 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:07.065299 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:07.065398 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:07.065380 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:09.065017 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.064932 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:09.065513 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:09.065098 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:09.065513 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.065165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:09.065513 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:09.065235 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:09.065513 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.065281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:09.065513 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:09.065360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:09.107573 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.107524 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="aa6b89951836e6968e213095ca4aef2a621ef3d18f4897807770c53b2298f949" exitCode=0 Nov 03 01:17:09.107803 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.107646 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"aa6b89951836e6968e213095ca4aef2a621ef3d18f4897807770c53b2298f949"} Nov 03 01:17:09.115058 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.115033 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"b5ac7307a30e6ec216df9c122c45451f","Type":"ContainerStarted","Data":"526794604f5c99621cb9ba8aaeb77563646274f6d7aba9bc507887eb3a8ca17b"} Nov 03 01:17:09.123971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.123780 2573 generic.go:356] "Generic (PLEG): container finished" podID="bb5781414dea338e9e0fd077df61848e" containerID="fafd1b7b774ac9d37676bffe32f8dde006ec22ae18aa05fb3ada306d9b035f41" exitCode=0 Nov 03 01:17:09.123971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.123822 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerDied","Data":"fafd1b7b774ac9d37676bffe32f8dde006ec22ae18aa05fb3ada306d9b035f41"} Nov 03 01:17:09.564975 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:09.564910 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" podStartSLOduration=7.564892288 podStartE2EDuration="7.564892288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:09.564594724 +0000 UTC m=+7.974808328" watchObservedRunningTime="2026-11-03 01:17:09.564892288 +0000 UTC m=+7.975105892" Nov 03 01:17:10.130571 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.130516 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerStarted","Data":"64e6cd9aa3a0d259669215b337e578fa8b97da3d0c400068ec070dea0ca662ef"} Nov 03 01:17:10.139149 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.139122 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjn7g" event={"ID":"add38c69-7c29-44f2-940f-061783060dcd","Type":"ContainerStarted","Data":"986fde0d698b3f4a9326ce78a077b13cef26cacc665221c41efa5b27e67829a7"} Nov 03 01:17:10.246387 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.246051 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" podStartSLOduration=8.246031929 podStartE2EDuration="8.246031929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:10.245747375 +0000 UTC m=+8.655960980" watchObservedRunningTime="2026-11-03 01:17:10.246031929 +0000 UTC m=+8.656245538" Nov 03 01:17:10.629807 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.629725 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:10.633139 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.632745 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:10.748718 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.748377 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tjn7g" podStartSLOduration=3.84481558 podStartE2EDuration="8.748355454s" totalImagesPullingTime="4.903539874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:10.361424978 +0000 UTC m=+8.771638585" watchObservedRunningTime="2026-11-03 01:17:10.748355454 +0000 UTC m=+9.158569058" Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.959332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:10.959430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:10.959488 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:10.959535 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:10.959576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:18.959543604 +0000 UTC m=+17.369757186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:10.959621 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:10.959601 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:18.959585693 +0000 UTC m=+17.369799277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:11.060381 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.060345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:11.060595 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.060578 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:11.060657 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.060603 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:11.060657 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.060638 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:11.060762 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.060738 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:19.060717765 +0000 UTC m=+17.470931361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:11.064620 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.064593 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:11.064737 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.064719 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:11.064804 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.064790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:11.064896 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.064876 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:11.064954 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.064935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:11.065022 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:11.065004 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:11.148900 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.148822 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" event={"ID":"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa","Type":"ContainerStarted","Data":"a98f342028ea7d2982508ba519fff80cdd8573becf5b551a4c20f33f0862e993"} Nov 03 01:17:11.152305 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.152277 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkcgz" event={"ID":"43445eef-1fab-461d-a4db-bf26cb747247","Type":"ContainerStarted","Data":"21292bc97d291887ca9689602a02ca4df07e494a2e5e5d57604d3e1f1ae21d25"} Nov 03 01:17:11.267212 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:11.267152 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jkcgz" podStartSLOduration=3.667742597 podStartE2EDuration="9.267135439s" totalImagesPullingTime="5.599392842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:11.266886363 +0000 UTC m=+9.677099970" watchObservedRunningTime="2026-11-03 01:17:11.267135439 +0000 UTC m=+9.677349049" Nov 03 01:17:12.155248 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:12.155216 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.065006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:13.065132 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.065213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:13.065292 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.065363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:13.065478 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:13.065446 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:13.164588 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.163954 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vfccq" event={"ID":"31d87eeb-c7ba-4afa-bb30-9a512102f939","Type":"ContainerStarted","Data":"fb2f41b1e1917d6019d196e2ef0eb2ae4e3f7b0ba8e4edad1b068b3d02f04dc3"} Nov 03 01:17:13.171216 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.171044 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:17:13.282722 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:13.282268 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vfccq" podStartSLOduration=3.19744143 podStartE2EDuration="11.282255597s" totalImagesPullingTime="8.084814167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:13.282133462 +0000 UTC m=+11.692347057" watchObservedRunningTime="2026-11-03 01:17:13.282255597 +0000 UTC m=+11.692469200" Nov 03 01:17:14.075908 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.075800 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:17:13.171064256Z","UUID":"d6099b2d-b1d8-4313-b700-fda758db81eb","Handler":null,"Name":"","Endpoint":""} Nov 03 01:17:14.077873 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.077851 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:17:14.078003 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.077899 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:17:14.177550 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.177464 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerStarted","Data":"409cdad87b9ddf3ea5f4bcb84901b2cdf209514ac230ce6ac0326c025b18f0eb"} Nov 03 01:17:14.179890 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.179858 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" event={"ID":"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa","Type":"ContainerStarted","Data":"6d84601e27ac53dfda40a796bea661294c12c9aa1e235e486d26e28862d0f78d"} Nov 03 01:17:14.181896 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.181870 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rv9vt" event={"ID":"ac23fba7-e4fe-45bc-8360-9875079da1cd","Type":"ContainerStarted","Data":"f9b47831b0f6c9df7a67c502e13484b85260309ed6798a6e19cd0063d146b4a1"} Nov 03 01:17:14.430900 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:14.430439 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rv9vt" podStartSLOduration=3.969252595 podStartE2EDuration="12.430425244s" totalImagesPullingTime="8.461172649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:14.429755576 +0000 UTC m=+12.839969181" watchObservedRunningTime="2026-11-03 01:17:14.430425244 +0000 UTC m=+12.840638847" Nov 03 01:17:15.064972 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.064944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:15.065086 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.065042 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:15.065213 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.065196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:15.065256 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:15.065223 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:15.065313 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:15.065296 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:15.065372 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:15.065357 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:15.185312 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.185225 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d64cz" event={"ID":"91186c34-ad26-42d9-a4e4-70f9da7a10db","Type":"ContainerStarted","Data":"5affe02a597f7a456d8eb519b2b8aa9c0d66338081db75b4ad51b820d93417eb"} Nov 03 01:17:15.187028 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.187000 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wq78" event={"ID":"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8","Type":"ContainerStarted","Data":"d659e6a57aab0f33a528c6a70f383966ac3f5f5653465bbbce8dd6ef0e61520e"} Nov 03 01:17:15.189110 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.189083 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="409cdad87b9ddf3ea5f4bcb84901b2cdf209514ac230ce6ac0326c025b18f0eb" exitCode=0 Nov 03 01:17:15.189221 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.189175 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"409cdad87b9ddf3ea5f4bcb84901b2cdf209514ac230ce6ac0326c025b18f0eb"} Nov 03 01:17:15.191833 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.191814 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" event={"ID":"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa","Type":"ContainerStarted","Data":"ad02d486244a8823b7f326403f7296961d8b99b7b76383c27100fa3bf303a612"} Nov 03 01:17:15.303080 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.303030 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d64cz" podStartSLOduration=2.874080426 podStartE2EDuration="13.303016435s" totalImagesPullingTime="10.428936009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:15.302442476 +0000 UTC m=+13.712656082" watchObservedRunningTime="2026-11-03 01:17:15.303016435 +0000 UTC m=+13.713230040" Nov 03 01:17:15.418246 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.418181 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9wq78" podStartSLOduration=4.946178816 podStartE2EDuration="13.41816099s" totalImagesPullingTime="8.471982174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:15.418017926 +0000 UTC m=+13.828231531" watchObservedRunningTime="2026-11-03 01:17:15.41816099 +0000 UTC m=+13.828374595" Nov 03 01:17:15.682448 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:15.682352 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" podStartSLOduration=3.695224912 podStartE2EDuration="13.682333687s" totalImagesPullingTime="9.987108775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:15.682300943 +0000 UTC m=+14.092514547" watchObservedRunningTime="2026-11-03 01:17:15.682333687 +0000 UTC m=+14.092547289" Nov 03 01:17:17.064656 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:17.064579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:17.065282 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:17.064589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:17.065282 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:17.064700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:17.065282 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:17.064589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:17.065282 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:17.064775 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:17.065282 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:17.064869 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:17.198469 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:17.198434 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="316f0a31011564fbcdc99c26b4d96d8fd29b11145b158f535b9283a07b8be6b1" exitCode=0 Nov 03 01:17:17.198667 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:17.198483 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"316f0a31011564fbcdc99c26b4d96d8fd29b11145b158f535b9283a07b8be6b1"} Nov 03 01:17:19.019248 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.018951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:19.019248 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.019208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:19.019248 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.019110 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:19.019741 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.019310 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.019292061 +0000 UTC m=+33.429505647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:19.019741 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.019333 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:19.019741 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.019378 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret podName:cbd9a86f-c50f-41b2-9c20-fe94fda357c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.019367318 +0000 UTC m=+33.429580904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret") pod "global-pull-secret-syncer-xqklw" (UID: "cbd9a86f-c50f-41b2-9c20-fe94fda357c1") : object "kube-system"/"original-pull-secret" not registered Nov 03 01:17:19.065088 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.065053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:19.065248 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.065053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:19.065248 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.065192 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:19.065361 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.065257 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:19.065361 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.065054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:19.065361 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.065325 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:19.120300 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.120265 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:19.120459 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.120433 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 01:17:19.120530 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.120460 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 01:17:19.120530 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.120473 2573 projected.go:196] Error preparing data for projected volume kube-api-access-xn8qt for pod openshift-network-diagnostics/network-check-target-qk55d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:19.120530 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:19.120527 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt podName:e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.120509698 +0000 UTC m=+33.530723297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xn8qt" (UniqueName: "kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt") pod "network-check-target-qk55d" (UID: "e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 01:17:19.205603 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.205571 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="5a9c670559fd7484730ac35ecf63801128e455837cb657e5fe22489dc188ec99" exitCode=0 Nov 03 01:17:19.205750 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:19.205614 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"5a9c670559fd7484730ac35ecf63801128e455837cb657e5fe22489dc188ec99"} Nov 03 01:17:21.065474 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.065293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:21.065901 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.065339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:21.065901 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:21.065588 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:21.065901 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.065351 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:21.065901 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:21.065650 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:21.065901 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:21.065699 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:21.213032 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.212959 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"7ed4d7cbda6c4ea7e7eea6ceba93e715305a3cd3699a84be641b428cd5174c43"} Nov 03 01:17:21.213032 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.213027 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"20b6560a7aff80e0c1d8b8a6021e7d09b9d20e356342c19e8a966d7ae14d9c32"} Nov 03 01:17:21.213190 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.213058 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"098e3cffcd0b5b5e80f5e6e04fa1e4c6372c67c09f14b555f112273e519b9491"} Nov 03 01:17:21.213190 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.213086 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"d39ce0196244a4603d6c5b1d7220cd745cfcad0718626d800f1e0fde40261a56"} Nov 03 01:17:21.213190 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.213112 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"90772d5f236b946464b7cfc927dac23930a149c824a0ce5351c1c508b69ba396"} Nov 03 01:17:21.213190 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:21.213140 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"397b9b4a996ecd4aaf9690a41e347c8070ecb648023c769c4dcf91e712107f6d"} Nov 03 01:17:23.064625 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.064594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:23.065168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.064646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:23.065168 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.064675 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:23.065168 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:23.064764 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:23.065168 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:23.064807 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:23.065168 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:23.064828 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:23.220293 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.220262 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"beeb0adcc12e6ace7a647b3d880975ba21498954dff942e2773d4d34adfec22e"} Nov 03 01:17:23.222737 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.222711 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="69e5697e8228e693005d6b6bd29b7fc9e16c4eb0fca43732a955dbc3b1023f64" exitCode=0 Nov 03 01:17:23.222828 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:23.222757 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"69e5697e8228e693005d6b6bd29b7fc9e16c4eb0fca43732a955dbc3b1023f64"} Nov 03 01:17:24.228156 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:24.228122 2573 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="315b7228d65dcf7770087697e28ca3df45c819613912171d711d9bd5001dfda5" exitCode=0 Nov 03 01:17:24.228536 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:24.228178 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerDied","Data":"315b7228d65dcf7770087697e28ca3df45c819613912171d711d9bd5001dfda5"} Nov 03 01:17:25.064798 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.064777 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:25.064920 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.064867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:25.064920 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:25.064873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:25.064995 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:25.064947 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:25.064995 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.064973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:25.065067 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:25.065056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:25.237813 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.237773 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" event={"ID":"3deae577-9838-4896-82bf-a1b5b5710d3a","Type":"ContainerStarted","Data":"f6ee949323a8bdc68caec857dc2cc42be4132609ec35b4639af39f03058c77bd"} Nov 03 01:17:25.238688 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.238064 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:25.238688 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.238132 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:25.242525 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.242498 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" event={"ID":"954cc14b-f026-4cf9-93cb-b14e3b7ef933","Type":"ContainerStarted","Data":"710193c805d9a81f3a9debccf2802a0eda26b8e3d18a42429191c9a50470dbdf"} Nov 03 01:17:25.254886 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.254862 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:25.255016 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.254957 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:25.448723 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.448456 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" podStartSLOduration=7.517879235 podStartE2EDuration="23.448443849s" totalImagesPullingTime="15.930564614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:25.448074649 +0000 UTC m=+23.858288253" watchObservedRunningTime="2026-11-03 01:17:25.448443849 +0000 UTC m=+23.858657501" Nov 03 01:17:25.573321 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:25.573274 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6g9zq" podStartSLOduration=9.186013427 podStartE2EDuration="23.573260358s" totalImagesPullingTime="14.387246931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:25.573079974 +0000 UTC m=+23.983293578" watchObservedRunningTime="2026-11-03 01:17:25.573260358 +0000 UTC m=+23.983473962" Nov 03 01:17:26.244631 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:26.244600 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:27.064974 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.064939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:27.065153 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.065001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:27.065153 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.065096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:27.065153 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.065098 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:27.065271 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.065167 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:27.065271 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.065227 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:27.247509 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.247476 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:27.787867 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.787833 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqklw"] Nov 03 01:17:27.788059 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.788009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:27.788155 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.788134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:27.791842 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.791820 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk55d"] Nov 03 01:17:27.791948 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.791914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:27.791994 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.791979 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:27.838150 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.838122 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9lj4"] Nov 03 01:17:27.838284 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:27.838224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:27.838328 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:27.838311 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:29.051525 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:29.051494 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:29.051906 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:29.051619 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:29.052109 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:29.052093 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:29.065193 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:29.065165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:29.065322 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:29.065169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:29.065322 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:29.065277 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:29.065393 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:29.065330 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:30.067182 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:30.067115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:30.067538 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:30.067207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:31.064962 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:31.064925 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:31.065148 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:31.065092 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:31.065148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:31.065125 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:31.065283 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:31.065203 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:32.065911 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:32.065801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:32.066361 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:32.065952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqklw" podUID="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" Nov 03 01:17:33.064432 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.064400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:33.064612 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:33.064506 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x9lj4" podUID="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" Nov 03 01:17:33.064612 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.064576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:33.064708 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:33.064663 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk55d" podUID="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" Nov 03 01:17:33.380243 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.380214 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:33.380661 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.380440 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:33.398545 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.398515 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:33.891958 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.891928 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeReady" Nov 03 01:17:33.892114 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.892046 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 01:17:33.979433 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:33.979402 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kgrh"] Nov 03 01:17:34.000945 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.000916 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7rq8q"] Nov 03 01:17:34.001110 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.001058 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:34.020983 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.020957 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pm5n"] Nov 03 01:17:34.021136 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.021079 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:34.043156 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.043130 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg"] Nov 03 01:17:34.043286 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.043242 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:34.053566 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.053531 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.053664 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.053629 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.067360 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.067336 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.067479 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.067400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:34.071047 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.071026 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85ff878dc9-x2zvj"] Nov 03 01:17:34.097821 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.097792 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9"] Nov 03 01:17:34.098014 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.097998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.116318 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.115972 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvs7q"] Nov 03 01:17:34.116318 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.116087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:34.122309 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cs85\" (UniqueName: \"kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85\") pod \"kas-connection-checker-685895c7c9-5kgrh\" (UID: \"57f9e78d-c5d8-4af3-bbbb-15fddc658e70\") " pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:34.122441 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122316 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwq4\" (UniqueName: \"kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4\") pod \"kas-connection-checker-685895c7c9-6pm5n\" (UID: \"88af5e8b-13e7-47c8-9c9d-4c861f7c340b\") " pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:34.122441 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lst7s\" (UniqueName: \"kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s\") pod \"kas-connection-checker-685895c7c9-7rq8q\" (UID: \"8e263b05-c71a-4d41-99d7-209ec4dbad6a\") " pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:34.122441 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122382 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 01:17:34.122681 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122664 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:17:34.122768 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.122720 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\"" Nov 03 01:17:34.142150 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.142078 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp"] Nov 03 01:17:34.142301 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.142212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.154006 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.153981 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.154006 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.153989 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.154380 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.154351 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 01:17:34.154510 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.154496 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.154602 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.154586 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.154733 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.154719 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 01:17:34.168744 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.168724 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.168859 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.168846 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.171773 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.171757 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jftg"] Nov 03 01:17:34.171899 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.171884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.194425 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.194404 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm"] Nov 03 01:17:34.194543 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.194494 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.194981 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.194952 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 01:17:34.195089 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.194951 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.195270 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.195187 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.199210 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.199193 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 01:17:34.216733 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.216696 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln"] Nov 03 01:17:34.216875 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.216812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.222804 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cs85\" (UniqueName: \"kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85\") pod \"kas-connection-checker-685895c7c9-5kgrh\" (UID: \"57f9e78d-c5d8-4af3-bbbb-15fddc658e70\") " pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:34.222913 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwq4\" (UniqueName: \"kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4\") pod \"kas-connection-checker-685895c7c9-6pm5n\" (UID: \"88af5e8b-13e7-47c8-9c9d-4c861f7c340b\") " pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:34.222913 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.222913 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4st\" (UniqueName: \"kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqp2\" (UniqueName: \"kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2\") pod \"volume-data-source-validator-58c9f5f46d-jldp9\" (UID: \"b871da5c-3f46-4189-ab3c-b8f94f9111d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222948 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222957 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.222976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst7s\" (UniqueName: \"kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s\") pod \"kas-connection-checker-685895c7c9-7rq8q\" (UID: \"8e263b05-c71a-4d41-99d7-209ec4dbad6a\") " pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:34.223092 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223106 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223115 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2m9b\" (UniqueName: \"kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.223373 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223354 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvpm\" (UniqueName: \"kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.223683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.223607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.224228 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.224210 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 01:17:34.232837 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.232820 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:17:34.232959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.232943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.249036 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.249011 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4frnh"] Nov 03 01:17:34.249136 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.249121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.253968 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.253948 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.254119 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.254100 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 01:17:34.254192 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.254108 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.254192 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.254154 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 01:17:34.254192 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.254174 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.254341 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.254269 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.255121 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.255101 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 01:17:34.265176 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.265158 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:17:34.265268 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.265257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.284468 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.284446 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwq4\" (UniqueName: \"kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4\") pod \"kas-connection-checker-685895c7c9-6pm5n\" (UID: \"88af5e8b-13e7-47c8-9c9d-4c861f7c340b\") " pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:34.284576 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.284449 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cs85\" (UniqueName: \"kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85\") pod \"kas-connection-checker-685895c7c9-5kgrh\" (UID: \"57f9e78d-c5d8-4af3-bbbb-15fddc658e70\") " pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:34.284576 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.284497 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst7s\" (UniqueName: \"kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s\") pod \"kas-connection-checker-685895c7c9-7rq8q\" (UID: \"8e263b05-c71a-4d41-99d7-209ec4dbad6a\") " pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:34.291420 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.291402 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6"] Nov 03 01:17:34.291525 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.291510 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.317791 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.317767 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ms9cn"] Nov 03 01:17:34.317922 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.317895 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:34.323189 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323169 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 01:17:34.323849 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323179 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 01:17:34.323849 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323236 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323236 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323373 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323385 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323390 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323446 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8lzdq\"" Nov 03 01:17:34.324002 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.323454 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh6sf\"" Nov 03 01:17:34.324207 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.324207 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.324207 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.324350 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.324253 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:34.324350 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.324350 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.324323 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.824308135 +0000 UTC m=+33.234521716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:34.324515 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.324515 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvpm\" (UniqueName: \"kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.324515 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.324515 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324455 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vsq\" (UniqueName: \"kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.324515 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.324706 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqp2\" (UniqueName: \"kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2\") pod \"volume-data-source-validator-58c9f5f46d-jldp9\" (UID: \"b871da5c-3f46-4189-ab3c-b8f94f9111d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:34.324744 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324714 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwj4\" (UniqueName: \"kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.324782 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.324826 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2m9b\" (UniqueName: \"kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.324826 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.324962 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.325013 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.324975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.325054 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb69j\" (UniqueName: \"kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.325093 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.325060 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:34.325125 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.325113 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.825097307 +0000 UTC m=+33.235310904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:34.325185 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325163 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.325239 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.325305 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325285 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.325355 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bxt\" (UniqueName: \"kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.325424 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.325463 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.325504 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.325483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.325928 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.325914 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.825492701 +0000 UTC m=+33.235706283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:34.326017 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326000 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.326048 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.326095 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4st\" (UniqueName: \"kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.326128 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.326164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326130 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.326164 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.326254 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326177 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.326254 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.326254 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.326397 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.326397 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.326397 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326321 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.326397 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.326397 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.326636 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.326507 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.327121 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327099 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.327237 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327171 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.327453 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327428 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.327601 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327580 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.327841 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:34.328150 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.327624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.329409 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.329382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.332571 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.332527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.342119 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.342095 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rrblm"] Nov 03 01:17:34.342211 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.342172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.347428 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.347408 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:34.354167 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354147 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.354255 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354171 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.354255 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354185 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.354255 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354199 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 01:17:34.354255 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354234 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.354255 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354235 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 01:17:34.354428 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.354311 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 01:17:34.355125 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.355112 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 01:17:34.360410 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360389 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7rq8q"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360420 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pm5n"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360438 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kgrh"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360451 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360464 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360478 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvs7q"] Nov 03 01:17:34.360490 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360492 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jftg"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360504 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360517 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4frnh"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360530 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360542 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ms9cn"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360580 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360601 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360618 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85ff878dc9-x2zvj"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360632 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360645 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrblm"] Nov 03 01:17:34.360677 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.360656 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:17:34.378055 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.378031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:34.385612 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.385583 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvpm\" (UniqueName: \"kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.386459 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.386432 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2m9b\" (UniqueName: \"kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.386647 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.386630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqp2\" (UniqueName: \"kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2\") pod \"volume-data-source-validator-58c9f5f46d-jldp9\" (UID: \"b871da5c-3f46-4189-ab3c-b8f94f9111d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:34.387027 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.387010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4st\" (UniqueName: \"kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.423361 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.423201 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:17:34.423361 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.423308 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\"" Nov 03 01:17:34.423504 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.423372 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\"" Nov 03 01:17:34.423504 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.423405 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:17:34.427626 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.427688 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.427688 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.427688 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.427772 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.427772 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.427772 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.427866 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vsq\" (UniqueName: \"kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.427866 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.427866 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwj4\" (UniqueName: \"kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.427971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.427971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.427971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.427971 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.428159 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.427985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.428821 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb69j\" (UniqueName: \"kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.428821 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.428816 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:34.428959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.428959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.428959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.428959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.429148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.429148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.428996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.429148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.429148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bxt\" (UniqueName: \"kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.429148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6z65\" (UniqueName: \"kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknc5\" (UniqueName: \"kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5\") pod \"network-check-source-6f78d59f99-9rfn6\" (UID: \"080c75f8-1ec9-432b-aede-34b6d715ab6f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.429364 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.429584 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.429436 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.929418636 +0000 UTC m=+33.339632232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:34.429584 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.430019 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.429993 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.430127 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430053 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.430127 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.430359 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.430297 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:34.430428 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.430378 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.930360709 +0000 UTC m=+33.340574304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:34.430428 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430384 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.430529 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.430600 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.430660 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.430660 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.430747 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-259tj\" (UniqueName: \"kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.430933 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.430911 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:34.431005 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.430938 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:34.431005 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.430977 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:34.930963591 +0000 UTC m=+33.341177192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:34.431005 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.430648 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.431301 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.431277 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.431772 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.431744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.431846 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.431782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.431994 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.431971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.432414 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.432392 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.432821 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.432796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.433095 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.433058 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.433533 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.433504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.434426 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.434396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.457623 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.456737 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:17:34.457623 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.456920 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.457623 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.457018 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.457623 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.457161 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:17:34.457623 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.457245 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:17:34.463923 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.463758 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:34.485065 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.485031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:34.491162 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.491136 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vsq\" (UniqueName: \"kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.491293 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.491135 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.492165 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.492143 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.492242 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.492191 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb69j\" (UniqueName: \"kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.493378 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.493359 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwj4\" (UniqueName: \"kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.493941 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.493922 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bxt\" (UniqueName: \"kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.509019 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.508987 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:34.527825 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.527472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:34.533017 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.532986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533107 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533173 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259tj\" (UniqueName: \"kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.533173 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.533287 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533383 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.533383 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533509 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533509 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.533509 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533509 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.533649 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.533649 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6z65\" (UniqueName: \"kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.533649 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.533624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknc5\" (UniqueName: \"kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5\") pod \"network-check-source-6f78d59f99-9rfn6\" (UID: \"080c75f8-1ec9-432b-aede-34b6d715ab6f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:34.534153 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.534130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.534463 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.534439 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.534586 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.534541 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:34.534586 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.534577 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:34.534660 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.534627 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.034610214 +0000 UTC m=+33.444823797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:34.534879 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.534857 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.535905 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.535244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.535905 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.535354 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.535905 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.535812 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:34.535905 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.535865 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.035848733 +0000 UTC m=+33.446062315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:34.537353 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.537328 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.538611 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.538081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.541998 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.541974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.597541 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.596283 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.598616 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.598265 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknc5\" (UniqueName: \"kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5\") pod \"network-check-source-6f78d59f99-9rfn6\" (UID: \"080c75f8-1ec9-432b-aede-34b6d715ab6f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:34.599874 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.598735 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259tj\" (UniqueName: \"kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:34.599874 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.599833 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6z65\" (UniqueName: \"kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.599874 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.599850 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:34.604480 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.604455 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:34.709057 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.709014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:34.733977 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.733953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:34.780504 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.780396 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7rq8q"] Nov 03 01:17:34.786693 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.782055 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kgrh"] Nov 03 01:17:34.786693 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.783009 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6pm5n"] Nov 03 01:17:34.836043 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.836013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:34.836393 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.836293 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:34.836600 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.836587 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.836541107 +0000 UTC m=+34.246754695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:34.836812 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.836797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.837054 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.836961 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:34.837180 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.837170 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.837155935 +0000 UTC m=+34.247369518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:34.837359 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.837345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:34.837578 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.837533 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.83751896 +0000 UTC m=+34.247732558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:34.850292 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.850262 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9"] Nov 03 01:17:34.851249 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:34.851222 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb871da5c_3f46_4189_ab3c_b8f94f9111d9.slice/crio-aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1 WatchSource:0}: Error finding container aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1: Status 404 returned error can't find the container with id aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1 Nov 03 01:17:34.887136 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.887049 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tvs7q"] Nov 03 01:17:34.892218 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:34.892154 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacd2582_2d59_4dc3_ba1f_28118cb27609.slice/crio-925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428 WatchSource:0}: Error finding container 925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428: Status 404 returned error can't find the container with id 925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428 Nov 03 01:17:34.908206 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.908125 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp"] Nov 03 01:17:34.910013 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:34.909987 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8895329f_6ed6_41c8_9f6b_d268dff1c4dc.slice/crio-5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c WatchSource:0}: Error finding container 5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c: Status 404 returned error can't find the container with id 5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c Nov 03 01:17:34.941343 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.941310 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:34.941453 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.941396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:34.941453 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.941435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:34.941615 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941515 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:34.941615 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941610 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:34.941727 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941636 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.941607723 +0000 UTC m=+34.351821322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:34.941727 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941659 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.941648621 +0000 UTC m=+34.351862247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:34.941727 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941694 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:34.941727 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941710 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:34.941954 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:34.941755 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:35.9417425 +0000 UTC m=+34.351956094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:34.943731 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:34.943708 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6jftg"] Nov 03 01:17:34.950249 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:34.946227 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bfbf47_81e9_4f0c_818c_965de586692b.slice/crio-d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de WatchSource:0}: Error finding container d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de: Status 404 returned error can't find the container with id d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de Nov 03 01:17:35.010683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.010649 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln"] Nov 03 01:17:35.010826 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:35.010681 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477e52dd_4c41_4415_ada1_414b13751aef.slice/crio-112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b WatchSource:0}: Error finding container 112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b: Status 404 returned error can't find the container with id 112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b Nov 03 01:17:35.042662 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.042632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:35.042806 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.042671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:35.042806 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.042701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:35.042806 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.042759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:35.042806 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042768 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042855 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:18:07.042814443 +0000 UTC m=+65.453028027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042856 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042871 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042888 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042914 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:36.042896747 +0000 UTC m=+34.453110346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:35.042989 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.042934 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:36.04292425 +0000 UTC m=+34.453137833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:35.044935 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.044912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:35.065148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.065120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:35.065148 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.065144 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:35.091847 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:35.091821 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080c75f8_1ec9_432b_aede_34b6d715ab6f.slice/crio-5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72 WatchSource:0}: Error finding container 5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72: Status 404 returned error can't find the container with id 5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72 Nov 03 01:17:35.092245 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.092227 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6"] Nov 03 01:17:35.117926 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.117895 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:17:35.118191 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.118170 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\"" Nov 03 01:17:35.118244 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.118170 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\"" Nov 03 01:17:35.136067 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.136036 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrblm"] Nov 03 01:17:35.136166 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:35.136108 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7d5877_a1e3_4c88_af99_bf4911871bbe.slice/crio-d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8 WatchSource:0}: Error finding container d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8: Status 404 returned error can't find the container with id d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8 Nov 03 01:17:35.144030 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.144010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:35.146452 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.146434 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:35.267587 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.267498 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrblm" event={"ID":"6b7d5877-a1e3-4c88-af99-bf4911871bbe","Type":"ContainerStarted","Data":"d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8"} Nov 03 01:17:35.268603 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.268579 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" event={"ID":"080c75f8-1ec9-432b-aede-34b6d715ab6f","Type":"ContainerStarted","Data":"5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72"} Nov 03 01:17:35.269695 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.269673 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" event={"ID":"8895329f-6ed6-41c8-9f6b-d268dff1c4dc","Type":"ContainerStarted","Data":"5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c"} Nov 03 01:17:35.270749 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.270720 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" event={"ID":"b871da5c-3f46-4189-ab3c-b8f94f9111d9","Type":"ContainerStarted","Data":"aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1"} Nov 03 01:17:35.271592 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.271574 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" event={"ID":"35bfbf47-81e9-4f0c-818c-965de586692b","Type":"ContainerStarted","Data":"d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de"} Nov 03 01:17:35.273109 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.273086 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" event={"ID":"88af5e8b-13e7-47c8-9c9d-4c861f7c340b","Type":"ContainerStarted","Data":"1e23ee608060267050e3abb72bbfb7612ab8f09a7fc6bc6c5dcebc7fe29a0f80"} Nov 03 01:17:35.273176 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.273123 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" event={"ID":"88af5e8b-13e7-47c8-9c9d-4c861f7c340b","Type":"ContainerStarted","Data":"84a416928176638d21638167cafe09f19dd1f41aa5a651c6cbe474269c395de0"} Nov 03 01:17:35.274433 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.274412 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" event={"ID":"8e263b05-c71a-4d41-99d7-209ec4dbad6a","Type":"ContainerStarted","Data":"5fc47b7e5d7f3773a8e2caac992d4560a38f7f8e94e9201d731acf02aae99bd2"} Nov 03 01:17:35.274514 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.274454 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" event={"ID":"8e263b05-c71a-4d41-99d7-209ec4dbad6a","Type":"ContainerStarted","Data":"63f3a294c9296403b4e782d7485ddf57a52dc377f56854e6d381bacd3aa24ab0"} Nov 03 01:17:35.275495 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.275479 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" event={"ID":"477e52dd-4c41-4415-ada1-414b13751aef","Type":"ContainerStarted","Data":"112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b"} Nov 03 01:17:35.276536 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.276504 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" event={"ID":"cacd2582-2d59-4dc3-ba1f-28118cb27609","Type":"ContainerStarted","Data":"925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428"} Nov 03 01:17:35.277869 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.277848 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" event={"ID":"57f9e78d-c5d8-4af3-bbbb-15fddc658e70","Type":"ContainerStarted","Data":"eb2dc3d99027f8a6893d6587061cf07698b6f8c81fbc5ab9bda75636740b8ebe"} Nov 03 01:17:35.277946 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.277879 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" event={"ID":"57f9e78d-c5d8-4af3-bbbb-15fddc658e70","Type":"ContainerStarted","Data":"15c0fa58242608ad3f570cc7ca89e3dc550cc52cd68843e8fbe99ffe7f1d7953"} Nov 03 01:17:35.316541 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.316515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:35.410988 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.410959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:35.690079 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:35.690053 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd9a86f_c50f_41b2_9c20_fe94fda357c1.slice/crio-67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94 WatchSource:0}: Error finding container 67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94: Status 404 returned error can't find the container with id 67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94 Nov 03 01:17:35.690206 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.690133 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqklw"] Nov 03 01:17:35.821539 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:35.821510 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d3daaa_f3ab_4d3f_97e8_71ac93ee40a9.slice/crio-5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483 WatchSource:0}: Error finding container 5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483: Status 404 returned error can't find the container with id 5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483 Nov 03 01:17:35.821738 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.821719 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk55d"] Nov 03 01:17:35.851828 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.851799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:35.851959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.851851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:35.851959 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.851906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:35.851959 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.851946 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:35.852053 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.852013 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.851995636 +0000 UTC m=+36.262209218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:35.852053 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.852017 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:35.852053 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.852031 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.852023482 +0000 UTC m=+36.262237065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:35.852053 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.852051 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.852040695 +0000 UTC m=+36.262254307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:35.952859 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.952773 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.952870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:35.952918 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.952927 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.952990 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.952975989 +0000 UTC m=+36.363189575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.953000 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.953015 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.953029 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.953056 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.953042571 +0000 UTC m=+36.363256158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:35.953066 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:35.953070 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:37.953063718 +0000 UTC m=+36.363277301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:36.054243 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.054206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:36.054507 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.054268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:36.054507 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:36.054377 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:36.054507 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:36.054440 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:38.054424623 +0000 UTC m=+36.464638225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:36.054507 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:36.054463 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:36.054507 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:36.054487 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:36.054793 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:36.054544 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:38.054527668 +0000 UTC m=+36.464741263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:36.087376 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.087330 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" podStartSLOduration=419.087316515 podStartE2EDuration="6m59.087316515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.086875587 +0000 UTC m=+34.497089190" watchObservedRunningTime="2026-11-03 01:17:36.087316515 +0000 UTC m=+34.497530119" Nov 03 01:17:36.206445 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.206351 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" podStartSLOduration=419.206337782 podStartE2EDuration="6m59.206337782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.205783648 +0000 UTC m=+34.615997252" watchObservedRunningTime="2026-11-03 01:17:36.206337782 +0000 UTC m=+34.616551382" Nov 03 01:17:36.282617 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.282550 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk55d" event={"ID":"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9","Type":"ContainerStarted","Data":"5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483"} Nov 03 01:17:36.284778 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.284748 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqklw" event={"ID":"cbd9a86f-c50f-41b2-9c20-fe94fda357c1","Type":"ContainerStarted","Data":"67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94"} Nov 03 01:17:36.326124 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:36.326055 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" podStartSLOduration=419.326034413 podStartE2EDuration="6m59.326034413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.324943038 +0000 UTC m=+34.735156645" watchObservedRunningTime="2026-11-03 01:17:36.326034413 +0000 UTC m=+34.736248018" Nov 03 01:17:37.877800 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.877761 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.877840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.877899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.878011 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.878058 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.878045337 +0000 UTC m=+40.288258919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.878107 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.878100406 +0000 UTC m=+40.288313992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.878141 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:37.878275 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.878161 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.878155359 +0000 UTC m=+40.288368940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:37.978544 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.978503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:37.978739 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.978617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:37.978739 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:37.978680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:37.978905 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.978876 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:37.978968 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.978942 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.978923753 +0000 UTC m=+40.389137354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:37.979034 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.979009 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:37.979082 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.979041 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.979030533 +0000 UTC m=+40.389244130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:37.979136 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.979103 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:37.979136 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.979115 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:37.979220 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:37.979146 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.979134322 +0000 UTC m=+40.389347917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:38.079796 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:38.079756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:38.079981 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:38.079959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:38.080105 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:38.080090 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:38.080162 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:38.080154 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.080136699 +0000 UTC m=+40.490350305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:38.080236 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:38.080224 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:38.080290 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:38.080240 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:38.080290 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:38.080275 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.080262744 +0000 UTC m=+40.490476341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:39.301215 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:39.301179 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" event={"ID":"b871da5c-3f46-4189-ab3c-b8f94f9111d9","Type":"ContainerStarted","Data":"6c8f12d5f6cc5329ef79fe48b5132b81886b5112d9e28aa6f5a5460227caa7dc"} Nov 03 01:17:40.310857 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:40.310769 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" event={"ID":"477e52dd-4c41-4415-ada1-414b13751aef","Type":"ContainerStarted","Data":"b5a22c92185ca93cc6d2aff327d8659b3519bbbe84c129469ec81ec6fb223095"} Nov 03 01:17:40.427714 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:40.427642 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" podStartSLOduration=389.305653753 podStartE2EDuration="6m34.427622275s" totalImagesPullingTime="5.121968522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:40.427609052 +0000 UTC m=+38.837822657" watchObservedRunningTime="2026-11-03 01:17:40.427622275 +0000 UTC m=+38.837835880" Nov 03 01:17:40.428317 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:40.428277 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" podStartSLOduration=391.017260981 podStartE2EDuration="6m34.428264423s" totalImagesPullingTime="3.411003442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:39.419705398 +0000 UTC m=+37.829919004" watchObservedRunningTime="2026-11-03 01:17:40.428264423 +0000 UTC m=+38.838478027" Nov 03 01:17:41.315461 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.315424 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" event={"ID":"cacd2582-2d59-4dc3-ba1f-28118cb27609","Type":"ContainerStarted","Data":"8492efaa6bd7ae9538de7e3e37ef7c24e8138bb2c4c173dd06c0c945ba45e8e7"} Nov 03 01:17:41.441834 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.440300 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" podStartSLOduration=389.831712624 podStartE2EDuration="6m35.440277551s" totalImagesPullingTime="5.608564927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:41.439747178 +0000 UTC m=+39.849960786" watchObservedRunningTime="2026-11-03 01:17:41.440277551 +0000 UTC m=+39.850491185" Nov 03 01:17:41.884960 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.884870 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz"] Nov 03 01:17:41.892724 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.892697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:41.893324 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.893297 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2"] Nov 03 01:17:41.900979 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.900948 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:41.925238 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.925205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6qg\" (UniqueName: \"kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg\") pod \"migrator-66df5d496b-8vdx2\" (UID: \"38ea1aba-6e28-4744-b0b2-af86439e36cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:41.925377 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.925259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:41.925377 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.925308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh5kw\" (UniqueName: \"kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw\") pod \"migrator-66df5d496b-bssmz\" (UID: \"5493a968-afc8-4a7d-aa18-22e2a14b8d70\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:41.925377 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.925351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:41.925530 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.925434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:41.925608 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:41.925579 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:41.925671 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:41.925640 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.925617219 +0000 UTC m=+48.335830804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:41.926139 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:41.926120 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:41.926217 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:41.926192 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.926175832 +0000 UTC m=+48.336389434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:41.926323 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:41.926273 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.926261388 +0000 UTC m=+48.336474974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:41.941014 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.940859 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz"] Nov 03 01:17:41.947289 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.947262 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:41.947483 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.947428 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2"] Nov 03 01:17:41.947679 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.947661 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wq28x\"" Nov 03 01:17:41.947768 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:41.947741 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 01:17:42.026501 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.026463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:42.026682 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.026543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6qg\" (UniqueName: \"kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg\") pod \"migrator-66df5d496b-8vdx2\" (UID: \"38ea1aba-6e28-4744-b0b2-af86439e36cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:42.026682 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.026618 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh5kw\" (UniqueName: \"kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw\") pod \"migrator-66df5d496b-bssmz\" (UID: \"5493a968-afc8-4a7d-aa18-22e2a14b8d70\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:42.026682 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.026643 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:42.026837 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.026715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:42.026888 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.026850 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:42.026888 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.026866 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:42.026974 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.026941 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:50.02692194 +0000 UTC m=+48.437135538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:42.027038 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.027013 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:42.027091 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.027049 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:50.027037481 +0000 UTC m=+48.437251077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:42.027541 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.027514 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:42.027541 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.027588 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:50.027571386 +0000 UTC m=+48.437784982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:42.057090 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.055542 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ktk4w"] Nov 03 01:17:42.060228 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.060197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.092898 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.092866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh5kw\" (UniqueName: \"kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw\") pod \"migrator-66df5d496b-bssmz\" (UID: \"5493a968-afc8-4a7d-aa18-22e2a14b8d70\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:42.100837 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.100805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6qg\" (UniqueName: \"kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg\") pod \"migrator-66df5d496b-8vdx2\" (UID: \"38ea1aba-6e28-4744-b0b2-af86439e36cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:42.114035 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.113981 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:17:42.114186 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.114093 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:17:42.114358 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.114337 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\"" Nov 03 01:17:42.114465 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.114450 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ktk4w"] Nov 03 01:17:42.127528 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.127683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.127683 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127593 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cgr\" (UniqueName: \"kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.127816 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:42.127872 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:42.127872 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.127872 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.127859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.128022 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.127995 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:42.128075 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.128032 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:50.128019894 +0000 UTC m=+48.538233488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:42.128140 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.128075 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:42.128140 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.128083 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:42.128140 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.128102 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:50.128096365 +0000 UTC m=+48.538309961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:42.213312 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.213046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:42.221642 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.221442 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.228847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.228891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.228972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.228992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.229013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cgr\" (UniqueName: \"kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.229606 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.229718 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.229762 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.729748303 +0000 UTC m=+41.139961898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.229809 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.229848 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 01:17:42.229938 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.229870 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:42.729863297 +0000 UTC m=+41.140076892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : configmap "kube-rbac-proxy" not found Nov 03 01:17:42.294620 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.294363 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cgr\" (UniqueName: \"kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.324075 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.322159 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-sdbcx-x69tk"] Nov 03 01:17:42.336788 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.336644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.343554 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.343516 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/0.log" Nov 03 01:17:42.343717 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.343592 2573 generic.go:356] "Generic (PLEG): container finished" podID="35bfbf47-81e9-4f0c-818c-965de586692b" containerID="e128ea325cafed606e03db46dbfe3e778f8bdfb0f816a0f10fecf1c980415e41" exitCode=255 Nov 03 01:17:42.344049 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.343988 2573 scope.go:122] "RemoveContainer" containerID="e128ea325cafed606e03db46dbfe3e778f8bdfb0f816a0f10fecf1c980415e41" Nov 03 01:17:42.344568 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.344212 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" event={"ID":"35bfbf47-81e9-4f0c-818c-965de586692b","Type":"ContainerDied","Data":"e128ea325cafed606e03db46dbfe3e778f8bdfb0f816a0f10fecf1c980415e41"} Nov 03 01:17:42.347641 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.347617 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrblm" event={"ID":"6b7d5877-a1e3-4c88-af99-bf4911871bbe","Type":"ContainerStarted","Data":"b4fa16cff23b0f09adca7e80e1a7569facdf48f36a2c5ad39d1f1a473d505be9"} Nov 03 01:17:42.380227 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.380106 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sdbcx-x69tk"] Nov 03 01:17:42.390157 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.390136 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-27r6g\"" Nov 03 01:17:42.434961 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.434938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.435109 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.435085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.435425 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.435302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.435987 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.435615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.536254 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.536254 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.536789 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.536789 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536766 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.536861 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.537018 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.536969 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.537472 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.537450 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.574392 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.572578 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rrblm" podStartSLOduration=3.270470382 podStartE2EDuration="9.572542593s" totalImagesPullingTime="6.302072211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:42.572237052 +0000 UTC m=+40.982450657" watchObservedRunningTime="2026-11-03 01:17:42.572542593 +0000 UTC m=+40.982756202" Nov 03 01:17:42.600160 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.600074 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz"] Nov 03 01:17:42.602548 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.602517 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.606580 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:42.606532 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5493a968_afc8_4a7d_aa18_22e2a14b8d70.slice/crio-e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2 WatchSource:0}: Error finding container e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2: Status 404 returned error can't find the container with id e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2 Nov 03 01:17:42.647196 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:42.647154 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ea1aba_6e28_4744_b0b2_af86439e36cb.slice/crio-91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff WatchSource:0}: Error finding container 91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff: Status 404 returned error can't find the container with id 91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff Nov 03 01:17:42.648553 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.648513 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2"] Nov 03 01:17:42.663897 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.663676 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:42.739059 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.739025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.739185 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:42.739161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:42.739290 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.739273 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 01:17:42.739358 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.739343 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:43.739322881 +0000 UTC m=+42.149536479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : configmap "kube-rbac-proxy" not found Nov 03 01:17:42.739446 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.739433 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:42.739497 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:42.739474 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:43.739461239 +0000 UTC m=+42.149674834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:43.041427 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.041131 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-sdbcx-x69tk"] Nov 03 01:17:43.042980 ip-10-0-138-19 kubenswrapper[2573]: W1103 01:17:43.042951 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefde973b_4882_427d_a08a_b0b14ef8aaa0.slice/crio-1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1 WatchSource:0}: Error finding container 1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1: Status 404 returned error can't find the container with id 1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1 Nov 03 01:17:43.352355 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.352285 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/1.log" Nov 03 01:17:43.352827 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.352733 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/0.log" Nov 03 01:17:43.352827 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.352759 2573 generic.go:356] "Generic (PLEG): container finished" podID="35bfbf47-81e9-4f0c-818c-965de586692b" containerID="711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228" exitCode=255 Nov 03 01:17:43.352827 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.352823 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" event={"ID":"35bfbf47-81e9-4f0c-818c-965de586692b","Type":"ContainerDied","Data":"711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228"} Nov 03 01:17:43.352960 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.352873 2573 scope.go:122] "RemoveContainer" containerID="e128ea325cafed606e03db46dbfe3e778f8bdfb0f816a0f10fecf1c980415e41" Nov 03 01:17:43.353670 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.353180 2573 scope.go:122] "RemoveContainer" containerID="711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228" Nov 03 01:17:43.353670 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:43.353386 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jftg_openshift-console-operator(35bfbf47-81e9-4f0c-818c-965de586692b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" podUID="35bfbf47-81e9-4f0c-818c-965de586692b" Nov 03 01:17:43.360080 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.360048 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk55d" event={"ID":"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9","Type":"ContainerStarted","Data":"315830a30c04b2799f1c75d586f89fad485706370f42c5bbe3b23e59c13da042"} Nov 03 01:17:43.360701 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.360683 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:43.362476 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.362432 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" event={"ID":"080c75f8-1ec9-432b-aede-34b6d715ab6f","Type":"ContainerStarted","Data":"4c0620e77dae0a85c3645ccbf883a7ee11e2551f5f05d9ed40c3c1c6ad2f824c"} Nov 03 01:17:43.364390 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.364360 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" event={"ID":"8895329f-6ed6-41c8-9f6b-d268dff1c4dc","Type":"ContainerStarted","Data":"3fe95386013e3b0c7dd144e2ecc6b771ba6f8a8667efd63780dd68e5da6e7927"} Nov 03 01:17:43.366190 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.366129 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" event={"ID":"38ea1aba-6e28-4744-b0b2-af86439e36cb","Type":"ContainerStarted","Data":"91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff"} Nov 03 01:17:43.367522 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.367480 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" event={"ID":"5493a968-afc8-4a7d-aa18-22e2a14b8d70","Type":"ContainerStarted","Data":"e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2"} Nov 03 01:17:43.369162 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.369139 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" event={"ID":"efde973b-4882-427d-a08a-b0b14ef8aaa0","Type":"ContainerStarted","Data":"bfee83737b05f00e80b08b4e1558da12333f438e68a5d8ff672e2498ab819e94"} Nov 03 01:17:43.369241 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.369180 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" event={"ID":"efde973b-4882-427d-a08a-b0b14ef8aaa0","Type":"ContainerStarted","Data":"1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1"} Nov 03 01:17:43.591300 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.588991 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" podStartSLOduration=389.755396671 podStartE2EDuration="6m37.588973318s" totalImagesPullingTime="7.833576647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:43.586465914 +0000 UTC m=+41.996679520" watchObservedRunningTime="2026-11-03 01:17:43.588973318 +0000 UTC m=+41.999186927" Nov 03 01:17:43.752895 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.752851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:43.753038 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:43.752931 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 01:17:43.753038 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.752971 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:43.753038 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:43.752996 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:45.752980305 +0000 UTC m=+44.163193891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : configmap "kube-rbac-proxy" not found Nov 03 01:17:43.753210 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:43.753050 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:43.753210 ip-10-0-138-19 kubenswrapper[2573]: E1103 01:17:43.753089 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:45.753078644 +0000 UTC m=+44.163292226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:43.820084 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.820034 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" podStartSLOduration=388.007623881 podStartE2EDuration="6m35.820020575s" totalImagesPullingTime="7.812396694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:43.819546085 +0000 UTC m=+42.229759689" watchObservedRunningTime="2026-11-03 01:17:43.820020575 +0000 UTC m=+42.230234178" Nov 03 01:17:43.820257 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.820137 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qk55d" podStartSLOduration=35.227463522 podStartE2EDuration="41.820131318s" totalImagesPullingTime="6.592667796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:43.701650657 +0000 UTC m=+42.111864263" watchObservedRunningTime="2026-11-03 01:17:43.820131318 +0000 UTC m=+42.230344914" Nov 03 01:17:43.936689 ip-10-0-138-19 kubenswrapper[2573]: I1103 01:17:43.936486 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" podStartSLOduration=1.93646581 podStartE2EDuration="1.93646581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:43.934734567 +0000 UTC m=+42.344948176" watchObservedRunningTime="2026-11-03 01:17:43.93646581 +0000 UTC m=+42.346679415" Nov 03 01:17:44.356643 ip-10-0-138-19 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 01:17:44.371062 ip-10-0-138-19 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 01:17:44.371292 ip-10-0-138-19 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 01:17:44.371351 ip-10-0-138-19 systemd[1]: kubelet.service: Consumed 3.115s CPU time. Nov 03 01:17:44.382156 ip-10-0-138-19 systemd[1]: Starting Kubernetes Kubelet... Nov 03 01:17:44.464545 ip-10-0-138-19 kubenswrapper[4992]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:44.464545 ip-10-0-138-19 kubenswrapper[4992]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 01:17:44.464545 ip-10-0-138-19 kubenswrapper[4992]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:44.464545 ip-10-0-138-19 kubenswrapper[4992]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467172 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467273 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467278 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467282 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467286 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467292 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.467286 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467296 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467301 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467305 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467310 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467314 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467317 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467320 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467324 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467328 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467333 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467340 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467344 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467350 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467354 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467359 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467364 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467368 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467375 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467382 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.467693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467386 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467392 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467398 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467403 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467408 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467412 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467418 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467422 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467427 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467431 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467435 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467438 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467441 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467444 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467447 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467450 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467457 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467462 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467466 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467471 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.468416 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467477 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467482 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467487 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467515 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467522 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467527 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467533 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467537 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467542 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467548 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467552 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467570 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467574 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467579 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467584 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467590 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467594 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467597 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467599 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467616 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.469181 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467622 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467626 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467633 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467638 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467643 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467648 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467651 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467654 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467657 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467660 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467662 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467665 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467668 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467671 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467673 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467676 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467679 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467681 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467684 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467687 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.469900 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467690 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467694 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467698 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467703 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467708 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467713 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467717 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467722 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467727 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467731 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467736 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467742 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467747 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467752 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467757 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467760 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467764 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467767 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467770 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.470606 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467773 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.467776 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468924 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468937 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468942 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468947 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468952 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468955 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468958 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468961 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468963 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468966 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468969 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468972 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468974 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468977 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468980 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468982 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468985 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468988 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.471173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468990 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468993 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.468997 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469000 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469003 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469010 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469018 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469023 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469029 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469035 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469040 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469045 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469048 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469050 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469053 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469056 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469058 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469061 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469064 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.472032 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469067 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469071 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469073 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469076 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469078 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469081 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469086 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469090 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469094 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469099 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469102 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469105 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469110 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469114 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469119 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469124 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469130 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469135 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469140 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469145 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.472844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469148 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469153 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469157 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469161 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469165 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469170 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469174 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469179 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469183 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469188 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469192 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469196 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469200 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469204 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469208 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469213 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469217 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469221 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469226 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.473722 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469229 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469234 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469238 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469241 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469246 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469251 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469256 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469261 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469264 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469267 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469275 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469280 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469284 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469288 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469292 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469296 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469300 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469304 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469309 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.474365 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469311 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469314 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469317 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469321 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469325 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469329 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469333 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469337 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469342 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469347 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.469352 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470067 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470076 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470079 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470082 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470085 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470087 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470090 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470093 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.474896 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470096 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470099 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470102 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470104 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470110 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470113 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470116 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470118 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470121 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470124 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470127 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470129 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470132 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470135 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470138 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470140 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470143 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470146 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470149 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470151 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.475433 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470154 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470157 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470159 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470162 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470165 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470167 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470170 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470173 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470177 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470181 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470185 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470189 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470194 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470198 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470202 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470206 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470212 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470217 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470222 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.476051 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470227 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470232 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470236 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470240 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470244 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470247 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470250 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470253 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470256 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470260 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470264 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470268 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470272 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470277 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470281 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470285 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470289 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470297 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470303 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.476641 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470308 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470312 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470317 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470322 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470326 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470329 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470332 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470334 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470338 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470341 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470345 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470348 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470351 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470355 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470358 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470361 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470363 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470366 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470369 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470373 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.477203 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470377 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470382 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470386 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470390 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470395 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470401 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470406 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470409 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470412 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470415 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470419 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470424 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470428 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470433 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470438 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470442 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470447 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470450 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470452 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.470458 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470591 4992 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 01:17:44.478045 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470601 4992 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470610 4992 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470617 4992 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470622 4992 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470627 4992 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470636 4992 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470640 4992 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470644 4992 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470647 4992 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470651 4992 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470657 4992 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470662 4992 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470667 4992 flags.go:64] FLAG: --cgroup-root="" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470672 4992 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470676 4992 flags.go:64] FLAG: --client-ca-file="" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470682 4992 flags.go:64] FLAG: --cloud-provider="external" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470685 4992 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470689 4992 flags.go:64] FLAG: --cluster-domain="" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470692 4992 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470695 4992 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470699 4992 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470702 4992 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470707 4992 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 01:17:44.478690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470710 4992 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470713 4992 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470717 4992 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470720 4992 flags.go:64] FLAG: --contention-profiling="false" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470723 4992 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470726 4992 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470730 4992 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470735 4992 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470741 4992 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470746 4992 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470752 4992 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470757 4992 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470761 4992 flags.go:64] FLAG: --enable-server="true" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470764 4992 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470769 4992 flags.go:64] FLAG: --event-burst="100" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470772 4992 flags.go:64] FLAG: --event-qps="50" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470775 4992 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470778 4992 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470782 4992 flags.go:64] FLAG: --eviction-hard="" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470786 4992 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470790 4992 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470793 4992 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470797 4992 flags.go:64] FLAG: --eviction-soft="" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470800 4992 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470803 4992 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 01:17:44.479424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470806 4992 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470810 4992 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470813 4992 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470816 4992 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470820 4992 flags.go:64] FLAG: --feature-gates="" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470827 4992 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470832 4992 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470837 4992 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470841 4992 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470844 4992 flags.go:64] FLAG: --healthz-port="10248" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470847 4992 flags.go:64] FLAG: --help="false" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470850 4992 flags.go:64] FLAG: --hostname-override="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470854 4992 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470857 4992 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470862 4992 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470865 4992 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470869 4992 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470873 4992 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470876 4992 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470880 4992 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470883 4992 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470886 4992 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470889 4992 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470892 4992 flags.go:64] FLAG: --kube-reserved="" Nov 03 01:17:44.480273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470895 4992 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470899 4992 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470904 4992 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470909 4992 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470913 4992 flags.go:64] FLAG: --lock-file="" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470919 4992 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470922 4992 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470925 4992 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470934 4992 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470939 4992 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470945 4992 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470950 4992 flags.go:64] FLAG: --logging-format="text" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470954 4992 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470958 4992 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470961 4992 flags.go:64] FLAG: --manifest-url="" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470965 4992 flags.go:64] FLAG: --manifest-url-header="" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470972 4992 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470977 4992 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470983 4992 flags.go:64] FLAG: --max-pods="110" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470987 4992 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470991 4992 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.470996 4992 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471002 4992 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471007 4992 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471010 4992 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 01:17:44.481211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471015 4992 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471022 4992 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471026 4992 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471029 4992 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471032 4992 flags.go:64] FLAG: --pod-cidr="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471035 4992 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471039 4992 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471042 4992 flags.go:64] FLAG: --pods-per-core="0" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471045 4992 flags.go:64] FLAG: --port="10250" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471048 4992 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471051 4992 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-00701afe6b45be27b" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471054 4992 flags.go:64] FLAG: --qos-reserved="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471057 4992 flags.go:64] FLAG: --read-only-port="10255" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471061 4992 flags.go:64] FLAG: --register-node="true" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471063 4992 flags.go:64] FLAG: --register-with-taints="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471068 4992 flags.go:64] FLAG: --registry-burst="10" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471072 4992 flags.go:64] FLAG: --registry-qps="5" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471076 4992 flags.go:64] FLAG: --reserved-cpus="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471081 4992 flags.go:64] FLAG: --reserved-memory="" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471087 4992 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471091 4992 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471094 4992 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471097 4992 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471100 4992 flags.go:64] FLAG: --runonce="false" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471103 4992 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471106 4992 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471109 4992 flags.go:64] FLAG: --seccomp-default="false" Nov 03 01:17:44.482526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471113 4992 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471116 4992 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471120 4992 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471124 4992 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471127 4992 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471132 4992 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471135 4992 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471138 4992 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471141 4992 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471144 4992 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471147 4992 flags.go:64] FLAG: --system-cgroups="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471151 4992 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471160 4992 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471165 4992 flags.go:64] FLAG: --tls-cert-file="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471169 4992 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471173 4992 flags.go:64] FLAG: --tls-min-version="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471176 4992 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471179 4992 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471182 4992 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471186 4992 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471189 4992 flags.go:64] FLAG: --v="2" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471194 4992 flags.go:64] FLAG: --version="false" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471199 4992 flags.go:64] FLAG: --vmodule="" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471203 4992 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.471207 4992 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 01:17:44.483734 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472070 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472083 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472089 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472093 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472098 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472102 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472107 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472122 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472150 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472157 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472163 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472168 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472174 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472179 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472184 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472189 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472194 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472199 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472204 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472216 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.484796 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472221 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472226 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472231 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472235 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472240 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472245 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472250 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472255 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472260 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472265 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472269 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472274 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472284 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472288 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472293 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472298 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472303 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472307 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472312 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472317 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.485407 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472322 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472327 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472332 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472336 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472341 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472351 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472356 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472360 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472365 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472370 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472377 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472385 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472390 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472395 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472400 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472406 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472410 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472419 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472424 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472428 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.486052 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472433 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472437 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472442 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472447 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472452 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472456 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472461 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472466 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472471 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472475 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472486 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472490 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472495 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472500 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472508 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472515 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472520 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472525 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.486784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472531 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472536 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472540 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472545 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472578 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472583 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472589 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472593 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472613 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472617 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472622 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472627 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472650 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472654 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472659 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472663 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472668 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472672 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472677 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472682 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.490376 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472687 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472691 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472696 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472700 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472705 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472849 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472854 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472858 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.472909 4992 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472988 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.472996 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473002 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473008 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473013 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473020 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473024 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.490980 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473028 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473033 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473037 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473042 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473046 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473050 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473055 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473059 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473064 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473068 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473073 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473077 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473082 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473086 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473091 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473095 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473100 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473105 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473109 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473113 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.491459 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473117 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473122 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473126 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473131 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473135 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473139 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473144 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473148 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473152 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473161 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473165 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473169 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473174 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473179 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473183 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473187 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473192 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473197 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473201 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473205 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.492002 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473210 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473215 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473219 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473224 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473229 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473233 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473238 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473242 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473246 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473251 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473255 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473260 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473264 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473269 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473273 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473278 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473282 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473286 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473290 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473294 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.492514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473299 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473305 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473310 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473314 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473318 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473322 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473327 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473331 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473335 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473340 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473346 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473353 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473359 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473363 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473370 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473374 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473378 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473382 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473387 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473391 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.493048 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473396 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473400 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473404 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473409 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473413 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473418 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473422 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473427 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473431 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473436 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473440 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473444 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473448 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473454 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473459 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473467 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473473 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473478 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.493802 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.473483 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.478014 4992 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.478036 4992 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.478059 4992 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.478065 4992 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478128 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478136 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478140 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478144 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478146 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478149 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478152 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478155 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478158 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478161 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478164 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478166 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478169 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478172 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478174 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.494629 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478177 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478180 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478183 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478186 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478189 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478191 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478194 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478197 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478201 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478206 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478211 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478214 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478217 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478220 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478225 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478228 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478230 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478233 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478236 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.495451 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478240 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478244 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478247 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478250 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478253 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478256 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478259 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478261 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478264 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478267 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478270 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478273 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478276 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478279 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478283 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478288 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478292 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478295 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478297 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478300 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.496288 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478303 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478305 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478309 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478314 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478318 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478321 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478324 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478326 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478330 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478334 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478337 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478339 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478342 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478345 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478348 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478350 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478354 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478358 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478363 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.497136 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478367 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478370 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478373 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478376 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478380 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478385 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478390 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478395 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478399 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478402 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478405 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478407 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478410 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478413 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478416 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478418 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478421 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478424 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478426 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478429 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.497978 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478433 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478437 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478443 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478446 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478449 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478452 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478455 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478458 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478460 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478463 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478466 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478469 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478471 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.478502 4992 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478677 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478684 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.498856 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478688 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478691 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478694 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478697 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478700 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478703 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478705 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478709 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478712 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478714 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478717 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478719 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478722 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478725 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478729 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478733 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478736 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478739 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478741 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.499512 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478747 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478750 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478753 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478755 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478758 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478761 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478763 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478766 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478768 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478771 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478774 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478776 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478780 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478784 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478788 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478793 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478796 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478799 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478802 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.500289 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478805 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478809 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478814 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478818 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478821 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478826 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478829 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478832 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478835 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478837 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478840 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478843 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478845 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478848 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478852 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478855 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478859 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478863 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478867 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.501111 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478872 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478875 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478877 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478880 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478883 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478885 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478888 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478891 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478893 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478896 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478898 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478901 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478904 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478906 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478909 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478912 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478914 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478917 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478920 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478923 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.501928 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478926 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478929 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478931 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478934 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478939 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478944 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478948 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478951 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478955 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478958 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478961 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478964 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478967 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478970 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478972 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478975 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478978 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478980 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478983 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478986 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.502548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478989 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478993 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.478997 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479001 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479006 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479010 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479014 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.479049 4992 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479089 4992 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479094 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479099 4992 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479102 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479105 4992 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479107 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479110 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479113 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 01:17:44.503083 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479115 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479118 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479121 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479123 4992 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479126 4992 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479129 4992 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479132 4992 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479135 4992 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479137 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479140 4992 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479143 4992 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479146 4992 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479148 4992 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479151 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479153 4992 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479158 4992 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479161 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479163 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479166 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 01:17:44.503478 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479169 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479171 4992 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479174 4992 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479178 4992 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479182 4992 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479186 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479191 4992 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479197 4992 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479203 4992 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479208 4992 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479214 4992 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479220 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479225 4992 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479230 4992 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479235 4992 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479239 4992 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479243 4992 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479248 4992 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479253 4992 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 01:17:44.504244 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479258 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479263 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479267 4992 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479271 4992 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479275 4992 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479280 4992 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479284 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479288 4992 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479291 4992 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479295 4992 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479298 4992 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479300 4992 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479303 4992 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479307 4992 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479309 4992 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479312 4992 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479315 4992 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479317 4992 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479320 4992 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479324 4992 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 01:17:44.505037 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479328 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479331 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479333 4992 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479336 4992 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479339 4992 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479342 4992 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479345 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479347 4992 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479351 4992 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479355 4992 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479360 4992 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479364 4992 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479368 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479373 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479378 4992 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479382 4992 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479387 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479391 4992 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479395 4992 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479400 4992 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 01:17:44.505857 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479405 4992 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479412 4992 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479415 4992 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479418 4992 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479421 4992 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479424 4992 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479427 4992 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479430 4992 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479432 4992 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479435 4992 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479438 4992 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479441 4992 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479447 4992 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479451 4992 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479454 4992 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479457 4992 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479460 4992 feature_gate.go:386] unrecognized feature gate: Example Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479465 4992 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479468 4992 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 01:17:44.506850 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:44.479471 4992 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.479538 4992 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.480803 4992 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.480872 4992 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.481100 4992 server.go:1014] "Starting client certificate rotation" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.481251 4992 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.481309 4992 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:02 +0000 UTC" deadline="2028-04-30 15:03:27.762971932 +0000 UTC" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.481352 4992 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13069h45m43.281623888s" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.481779 4992 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.483456 4992 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.485209 4992 log.go:25] "Validated CRI v1 runtime API" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.491587 4992 log.go:25] "Validated CRI v1 image API" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.492608 4992 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 01:17:44.507597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.495687 4992 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86498aeb-34e8-431d-813e-47863e9e50a1:/dev/nvme0n1p4 f787a52f-e48a-43b3-a9dd-a492aae411f6:/dev/nvme0n1p3] Nov 03 01:17:44.508168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.495714 4992 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15c0fa58242608ad3f570cc7ca89e3dc550cc52cd68843e8fbe99ffe7f1d7953/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15c0fa58242608ad3f570cc7ca89e3dc550cc52cd68843e8fbe99ffe7f1d7953/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63f3a294c9296403b4e782d7485ddf57a52dc377f56854e6d381bacd3aa24ab0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63f3a294c9296403b4e782d7485ddf57a52dc377f56854e6d381bacd3aa24ab0/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84a416928176638d21638167cafe09f19dd1f41aa5a651c6cbe474269c395de0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84a416928176638d21638167cafe09f19dd1f41aa5a651c6cbe474269c395de0/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a302280115504712a45c3a7f89cbb427ff50d5da18638af760b92d4b1777e20f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a302280115504712a45c3a7f89cbb427ff50d5da18638af760b92d4b1777e20f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa/volumes/kubernetes.io~projected/kube-api-access-vhbb8:{mountpoint:/var/lib/kubelet/pods/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa/volumes/kubernetes.io~projected/kube-api-access-vhbb8 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/080c75f8-1ec9-432b-aede-34b6d715ab6f/volumes/kubernetes.io~projected/kube-api-access-wknc5:{mountpoint:/var/lib/kubelet/pods/080c75f8-1ec9-432b-aede-34b6d715ab6f/volumes/kubernetes.io~projected/kube-api-access-wknc5 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d87eeb-c7ba-4afa-bb30-9a512102f939/volumes/kubernetes.io~projected/kube-api-access-s2kbl:{mountpoint:/var/lib/kubelet/pods/31d87eeb-c7ba-4afa-bb30-9a512102f939/volumes/kubernetes.io~projected/kube-api-access-s2kbl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~projected/kube-api-access-26bxt:{mountpoint:/var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~projected/kube-api-access-26bxt major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38df53d9-0193-43e6-a9a1-5ecd8fb2e456/volumes/kubernetes.io~projected/kube-api-access-j2cgr:{mountpoint:/var/lib/kubelet/pods/38df53d9-0193-43e6-a9a1-5ecd8fb2e456/volumes/kubernetes.io~projected/kube-api-access-j2cgr major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38ea1aba-6e28-4744-b0b2-af86439e36cb/volumes/kubernetes.io~projected/kube-api-access-jv6qg:{mountpoint:/var/lib/kubelet/pods/38ea1aba-6e28-4744-b0b2-af86439e36cb/volumes/kubernetes.io~projected/kube-api-access-jv6qg major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/kube-api-access-p96xj:{mountpoint:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/kube-api-access-p96xj major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4/volumes/kubernetes.io~projected/kube-api-access-6f4st:{mountpoint:/var/lib/kubelet/pods/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4/volumes/kubernetes.io~projected/kube-api-access-6f4st major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~projected/kube-api-access-2t67t:{mountpoint:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~projected/kube-api-access-2t67t major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43445eef-1fab-461d-a4db-bf26cb747247/volumes/kubernetes.io~projected/kube-api-access-tg6j4:{mountpoint:/var/lib/kubelet/pods/43445eef-1fab-461d-a4db-bf26cb747247/volumes/kubernetes.io~projected/kube-api-access-tg6j4 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~projected/kube-api-access-sb69j:{mountpoint:/var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~projected/kube-api-access-sb69j major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/502bb646-4048-4af0-90bb-a4f4688fe423/volumes/kubernetes.io~projected/kube-api-access-259tj:{mountpoint:/var/lib/kubelet/pods/502bb646-4048-4af0-90bb-a4f4688fe423/volumes/kubernetes.io~projected/kube-api-access-259tj major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5493a968-afc8-4a7d-aa18-22e2a14b8d70/volumes/kubernetes.io~projected/kube-api-access-qh5kw:{mountpoint:/var/lib/kubelet/pods/5493a968-afc8-4a7d-aa18-22e2a14b8d70/volumes/kubernetes.io~projected/kube-api-access-qh5kw major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57f9e78d-c5d8-4af3-bbbb-15fddc658e70/volumes/kubernetes.io~projected/kube-api-access-2cs85:{mountpoint:/var/lib/kubelet/pods/57f9e78d-c5d8-4af3-bbbb-15fddc658e70/volumes/kubernetes.io~projected/kube-api-access-2cs85 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bdb27b8-f440-43c6-a0fa-152432024154/volumes/kubernetes.io~projected/kube-api-access-47vsq:{mountpoint:/var/lib/kubelet/pods/5bdb27b8-f440-43c6-a0fa-152432024154/volumes/kubernetes.io~projected/kube-api-access-47vsq major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~projected/kube-api-access-v6z65:{mountpoint:/var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~projected/kube-api-access-v6z65 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~secret/cert major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8/volumes/kubernetes.io~projected/kube-api-access-mffq6:{mountpoint:/var/lib/kubelet/pods/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8/volumes/kubernetes.io~projected/kube-api-access-mffq6 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~projected/kube-api-access-wkvpm:{mountpoint:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~projected/kube-api-access-wkvpm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~projected/kube-api-access-xxwj4:{mountpoint:/var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~projected/kube-api-access-xxwj4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88af5e8b-13e7-47c8-9c9d-4c861f7c340b/volumes/kubernetes.io~projected/kube-api-access-2zwq4:{mountpoint:/var/lib/kubelet/pods/88af5e8b-13e7-47c8-9c9d-4c861f7c340b/volumes/kubernetes.io~projected/kube-api-access-2zwq4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e263b05-c71a-4d41-99d7-209ec4dbad6a/volumes/kubernetes.io~projected/kube-api-access-lst7s:{mountpoint:/var/lib/kubelet/pods/8e263b05-c71a-4d41-99d7-209ec4dbad6a/volumes/kubernetes.io~projected/kube-api-access-lst7s major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~projected/kube-api-access-p5j5g:{mountpoint:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~projected/kube-api-access-p5j5g major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954cc14b-f026-4cf9-93cb-b14e3b7ef933/volumes/kubernetes.io~projected/kube-api-access-r65hc:{mountpoint:/var/lib/kubelet/pods/954cc14b-f026-4cf9-93cb-b14e3b7ef933/volumes/kubernetes.io~projected/kube-api-access-r65hc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac23fba7-e4fe-45bc-8360-9875079da1cd/volumes/kubernetes.io~projected/kube-api-access-2dhc7:{mountpoint:/var/lib/kubelet/pods/ac23fba7-e4fe-45bc-8360-9875079da1cd/volumes/kubernetes.io~projected/kube-api-access-2dhc7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add38c69-7c29-44f2-940f-061783060dcd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/add38c69-7c29-44f2-940f-061783060dcd/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/kube-api-access-6nb5g:{mountpoint:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/kube-api-access-6nb5g major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b871da5c-3f46-4189-ab3c-b8f94f9111d9/volumes/kubernetes.io~projected/kube-api-access-qgqp2:{mountpoint:/var/lib/kubelet/pods/b871da5c-3f46-4189-ab3c-b8f94f9111d9/volumes/kubernetes.io~projected/kube-api-access-qgqp2 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~projected/kube-api-access-n2m9b:{mountpoint:/var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~projected/kube-api-access-n2m9b major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbd9a86f-c50f-41b2-9c20-fe94fda357c1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cbd9a86f-c50f-41b2-9c20-fe94fda357c1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9/volumes/kubernetes.io~projected/kube-api-access-xn8qt:{mountpoint:/var/lib/kubelet/pods/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9/volumes/kubernetes.io~projected/kube-api-access-xn8qt major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efde973b-4882-427d-a08a-b0b14ef8aaa0/volumes/kubernetes.io~projected/kube-api-access-wl8nn:{mountpoint:/var/lib/kubelet/pods/efde973b-4882-427d-a08a-b0b14ef8aaa0/volumes/kubernetes.io~projected/kube-api-access-wl8nn major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc/volumes/kubernetes.io~projected/kube-api-access-ld79v:{mountpoint:/var/lib/kubelet/pods/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc/volumes/kubernetes.io~projected/kube-api-access-ld79v major:0 minor:52 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6dd1ef8889b66790b4e57fc5d496b312893fcafaa106ced845f9afef7e6856c2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6418bf832645a4e62a1d09a90165873d5153b8b12e1aa1b5f6d61a84ada2923c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fe2d1df598f28d93e2419c173a3918521a75ad00915fc2932cd0db10d461eed7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/180d55bbd3ee9d4b8ac4baaad0d3d2697f790d300e614435347affc8f262b030/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/3ae97605c98ccb5c3e714aaaacc0fc92e5f652a47c11a4ad7d628bc386a0ed01/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fa941aa9682d3733be5e4992ac1c0a2e873ca5322134b5a5f1ffbc1dde156fa9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d692aa5f67211ef487d3f7342e34e874988b59586294fd166e7d38a09b9de86b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b308a1e9d1dde2fa73174d67f3833d4bea4a52018da03cfb96c12f3f0847e9fc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/638c880bf34f47785bbacec67dbf1243931499d869aeb2a1726d4db0896e2710/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1ccabf8a9d9fc4cdbe348629265e20c3cf66cd4c38d688fa17f94a80c40587f0/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5960e1b3aeae6965d87a788ec672ed52eb09d4e2e77d63ace34c5d244cb1d20e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f0fc9a7e976cdce27244e6ee3316979ff505f98abbf0365184fbb9c56b564bd0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/df386bbaf2651eb43d55c0a3c0066af5a5c7fa6a88ba412779186c80fd3063d3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a3a7be8aeca7952f5f3fe66aaddff7fa7153f318e216737989afd4e6d37f4c25/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e4c68d2bbd4c4933ef39e0230d351cbb1d567c2c1c3788e4cea1ee92049f8f8c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/55239d0ef5d9093da31f96db5c5b46c8c029dd4698b289c5a85a11889a79b4fa/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4b2f9fa0d1733a99e3df1815e7f99923010ca7034ec007c5f49f1a90c4726d23/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/06303031ec2b31d634afa78a374f2d3cf7da1f98273e5a4a9ebf4c2b39fbf05b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d0acfee734bbe9458ee90970bdec4e365a594399807b5b9fdf8d4fd204201920/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/73a21a5c12043171f909421faf57cf13c9e3fda879aaaf6aa064264fbee9c5d0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/05fb5364dde2102f98e5f46104de730ff38a10eeb4e52eb40d94467b39fee28b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9a2700f7ed833440a48f88b14843dc550a789cdc801a5dd368bf635e553f8aac/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b6c3d5255906f4256e3fbbd5f44db9315e7fca7d383e952748e1859aff84a7a9/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8da5a1379602acfe7196a2e1389f851a14229633a6906571a11b122c056e9600/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5fcf02bf770046511239b75709a59a5933619e87bf2258da2b0ab2e78392131d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8ab45f1f58d1a1da2c00ec7d89e5a9bf545179a24c73a06d8efa646f7b6896e7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/3730bc95e609dfeb1a4c67aa93eef84191aeddc3d7fd55537ba8416aeee0725a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/33d57a5e2ec6839527ac3c03b14c94e794bf384487b13611cd5f6e29ee3cb182/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/3934017de9d989e576e11cf6e4a531f57953083c00f12ab845a11f4aaab3a168/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f8000f0c64f8e849d2e39c16aa2931dca592fc02ec8d1f6053f7fd2ee449dc3d/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/dc77b800a0d0e00afcef573bbf197bc2039683a9c1ccf883a80bd11813062fa2/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/232c06d25c899d36bb9a586fa5c60db4a2b6a8c57111ba2edd16b9561c702993/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/04eaf27b56c4699dfe37e8436c967aade6ace28f6a4ef2aef585a4ecbafdd2e2/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/1215dc04efc802fe9c4abcb40ce16a5029c30367ef741635aff4371948e4a6aa/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/43351e8f5dbd7a897707908882fca51ccd337c07d88631b5ef9fcf1e5389f47b/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/a02ecd01cb44f98e809f862bf5f37c629742d55bb5de460187d86f140437a0e8/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/1a6e6111b9a708be11449b543776efc84ffad466f6de24e725a5f0713333d3e9/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3ef286b45d7349d49a743d2a863951837ac5666f27015c631730767c7e2d9626/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/91d31f5bfadaa08f4ef8bc71110051c49a519e85955bfa42296dea4531de9025/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/e32aff4b693c208c2e73937955749048254aa6219aff64ee1b320a3a183088f7/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/b7b42441b8e9a7e75344029083697185391057b4e61845e6ccfadc28246deadb/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/cf216336b41d0fd8179bf0e8bf8260b83a45d198bb37ebbd93da5ae2c65bc50e/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/941f23e77d93c2ad34f04da2382699a144d1e2780644858183ec5cd3c5789cf8/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/5cbc926b5ec016a3830ec328166f4223ddd6a21f0b095013615e30d773c50f8e/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/06533607d2fba5cf87d1505c620ed37dcde91b0a62bdfb0f4e80899688ae443d/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/4e39a1b5776608a16515f6d59afa96c62c88e95f7052c681fedf810d29a5aeee/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/2201654e17aaaa92cc705e3d005be83fa117ecac45c92686d8d7fde9b16019c8/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/27034fccddef7a885bf277502595062d4b4c49a6d639b724255017317a5167de/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/895ff81d5decaf8a3132773a566bd0e4cab7a816ef1f1eb9009770989904014f/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b0a295d068d5c334e37cfdd052e5db776626614f4e22059ecbbe048deb870606/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8e96449a45752403f1e29d1f3f7dbdd68c8310c80ce706ac239db5483df1f5cd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a8107adfb0a38d2b367e34adf17912b47b7197565d3176db7c688f4a31610430/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e1e862e1998b257569d4f4b80ee0197bc29d7f0a64f7ed5d5ad47f37d69ad001/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3c328ac13e0571c517b37f48d52a788d60941b3bf2ba10ad82c76cb576ae7719/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/35bd3681edd609f265cc56609ee0ea81fa0481d05bdfdb4f9a4ce0e90fa5f0ba/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/2431c9f98a8ad16f7b10d6c751a97bd9cec38207ca9ad5a1c4d11f9008826dda/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/869e76745da1ef74d46824acbda6755f7b2b42c8b9b626e94c2840bf7cbc5537/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0f9ee28d83cc93e4d3a7c902f411b605106e691b79820588dadd5b0d19eff8ca/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 01:17:44.508541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.507347 4992 manager.go:217] Machine: {Timestamp:2026-11-03 01:17:44.506540819 +0000 UTC m=+0.085363401 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099542 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e3a88865684c08a72dfbb2bca4d51 SystemUUID:ec2e3a88-8656-84c0-8a72-dfbb2bca4d51 BootID:786a1ea8-fd07-4a52-8c75-7cef5123c06b Filesystems:[{Device:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8/volumes/kubernetes.io~projected/kube-api-access-mffq6 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa/volumes/kubernetes.io~projected/kube-api-access-vhbb8 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0707cf5f1c80606473aa2227e828acf7808caa2e2e1c9b0b1e9060f5a3b1729/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e263b05-c71a-4d41-99d7-209ec4dbad6a/volumes/kubernetes.io~projected/kube-api-access-lst7s DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d3d9accd6b3f44752eaed214bedebbbac6f4edeb65ab26cd23b75c49139215c/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~projected/kube-api-access-v6z65 DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7130bb7fa7cacea3c2edf7c7cf80eb2936c4c9276856d52980861a2321eb6c2/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84a416928176638d21638167cafe09f19dd1f41aa5a651c6cbe474269c395de0/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~projected/kube-api-access-xxwj4 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b7d5877-a1e3-4c88-af99-bf4911871bbe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f5ea772a16edc1ba3996d18900feaf8bcc1c8d254ca84e902b79691d52b3c72/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~projected/kube-api-access-p5j5g DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efde973b-4882-427d-a08a-b0b14ef8aaa0/volumes/kubernetes.io~projected/kube-api-access-wl8nn DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/kube-api-access-6nb5g DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d44c3100eac1b60c2b159081fd1c664e2dcb5f37cb9d15a444b51432b625397/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15c0fa58242608ad3f570cc7ca89e3dc550cc52cd68843e8fbe99ffe7f1d7953/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~projected/kube-api-access-wkvpm DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/835f0a42-f920-4d77-9178-4a174fe526c1/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~projected/kube-api-access-n2m9b DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43445eef-1fab-461d-a4db-bf26cb747247/volumes/kubernetes.io~projected/kube-api-access-tg6j4 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/080c75f8-1ec9-432b-aede-34b6d715ab6f/volumes/kubernetes.io~projected/kube-api-access-wknc5 DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/kube-api-access-p96xj DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbd9a86f-c50f-41b2-9c20-fe94fda357c1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8895329f-6ed6-41c8-9f6b-d268dff1c4dc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38ea1aba-6e28-4744-b0b2-af86439e36cb/volumes/kubernetes.io~projected/kube-api-access-jv6qg DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38df53d9-0193-43e6-a9a1-5ecd8fb2e456/volumes/kubernetes.io~projected/kube-api-access-j2cgr DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~projected/kube-api-access-2t67t DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a302280115504712a45c3a7f89cbb427ff50d5da18638af760b92d4b1777e20f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa23ffe0c38c87b677a12e2075262335b6812760ed070f11bebcbf8dd842a8b1/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31d87eeb-c7ba-4afa-bb30-9a512102f939/volumes/kubernetes.io~projected/kube-api-access-s2kbl DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53635f1f483824d54b9b092815294c7607c02cc7048fe64047d76c785d4d941c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef43adf86d838cc08f6878c1f7b21f3cd9217ffe7806ac04870b3da0d616924d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6a6ce873ad8843ebd14c489a14dd476891f848c1336a1078555480b8d9848de/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88af5e8b-13e7-47c8-9c9d-4c861f7c340b/volumes/kubernetes.io~projected/kube-api-access-2zwq4 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/925f64151bbb4c362ceb20285893e1f02898f142f600bb0bf118a91a9fc23428/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5261856a57bbbe4795d4df0071441b050af60e60a5a3054b87748d13d5461483/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/502bb646-4048-4af0-90bb-a4f4688fe423/volumes/kubernetes.io~projected/kube-api-access-259tj DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/add38c69-7c29-44f2-940f-061783060dcd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b871da5c-3f46-4189-ab3c-b8f94f9111d9/volumes/kubernetes.io~projected/kube-api-access-qgqp2 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96f1fa6651369dbb66ef20d10a82ee2200faac32bd74eaf147e59a2c2df585a9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91186c34-ad26-42d9-a4e4-70f9da7a10db/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/954cc14b-f026-4cf9-93cb-b14e3b7ef933/volumes/kubernetes.io~projected/kube-api-access-r65hc DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57f9e78d-c5d8-4af3-bbbb-15fddc658e70/volumes/kubernetes.io~projected/kube-api-access-2cs85 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4/volumes/kubernetes.io~projected/kube-api-access-6f4st DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bdb27b8-f440-43c6-a0fa-152432024154/volumes/kubernetes.io~projected/kube-api-access-47vsq DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cacd2582-2d59-4dc3-ba1f-28118cb27609/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35bfbf47-81e9-4f0c-818c-965de586692b/volumes/kubernetes.io~projected/kube-api-access-26bxt DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3deae577-9838-4896-82bf-a1b5b5710d3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/112f6544cd157675367630f22055ed5ff9de7a8c9bfef025a7aee50885ad8f0b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d73c67521cf7693ad5f6a823a461aeb3bbfd86100211a5066e5a1bdbe1f419d8/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9/volumes/kubernetes.io~projected/kube-api-access-xn8qt DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5493a968-afc8-4a7d-aa18-22e2a14b8d70/volumes/kubernetes.io~projected/kube-api-access-qh5kw DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f33ba73dd34276d8eb8ba98bd350260a7cf174123d84099897864e42417b409/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc/volumes/kubernetes.io~projected/kube-api-access-ld79v DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/477e52dd-4c41-4415-ada1-414b13751aef/volumes/kubernetes.io~projected/kube-api-access-sb69j DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ac23fba7-e4fe-45bc-8360-9875079da1cd/volumes/kubernetes.io~projected/kube-api-access-2dhc7 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30992988942f497415cbdbd7fa4d74c518b43c6652d9cdbe862fd54f4777d712/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67c82b01367fed0b80a0d0b3087ef78f0b5f221e84fc793870f1e8493b892d94/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91dfd73fa5097a58b4bd875f082e3264652e83b1d3fc5c60cbf1a740c6986fff/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63f3a294c9296403b4e782d7485ddf57a52dc377f56854e6d381bacd3aa24ab0/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:112f6544cd15767 MacAddress:f2:54:c2:e8:a6:8a Speed:10000 Mtu:8901} {Name:15c0fa58242608a MacAddress:72:50:f5:4d:25:df Speed:10000 Mtu:8901} {Name:1f268fa63406db8 MacAddress:6e:be:68:c8:14:1a Speed:10000 Mtu:8901} {Name:5261856a57bbbe4 MacAddress:06:ab:eb:95:31:5a Speed:10000 Mtu:8901} {Name:5d3d9accd6b3f44 MacAddress:76:d0:96:7f:6e:7e Speed:10000 Mtu:8901} {Name:5f5ea772a16edc1 MacAddress:e2:10:b3:1d:e8:93 Speed:10000 Mtu:8901} {Name:63f3a294c929640 MacAddress:1a:23:ea:32:16:9c Speed:10000 Mtu:8901} {Name:67c82b01367fed0 MacAddress:12:aa:0c:8a:71:aa Speed:10000 Mtu:8901} {Name:84a416928176638 MacAddress:ee:9f:00:e2:52:76 Speed:10000 Mtu:8901} {Name:91dfd73fa5097a5 MacAddress:7e:fc:de:93:94:9e Speed:10000 Mtu:8901} {Name:925f64151bbb4c3 MacAddress:7a:2e:7a:09:10:37 Speed:10000 Mtu:8901} {Name:aa23ffe0c38c87b MacAddress:d6:da:69:ee:f1:c6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e2:da:d9:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:42:a6:95:9c:ec:bf Speed:0 Mtu:8901} {Name:d6a6ce873ad8843 MacAddress:52:23:a4:d2:73:a6 Speed:10000 Mtu:8901} {Name:d73c67521cf7693 MacAddress:9a:bb:fe:8f:60:16 Speed:10000 Mtu:8901} {Name:e7130bb7fa7cace MacAddress:26:cb:e0:6a:0d:a9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e2:da:d9:21 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:ae:d3:a2:7e:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 01:17:44.508541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.507953 4992 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 01:17:44.508541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508019 4992 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 01:17:44.508541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508345 4992 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508370 4992 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-19.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508600 4992 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508616 4992 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508636 4992 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508672 4992 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508703 4992 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 01:17:44.509015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.508942 4992 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509069 4992 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509107 4992 kubelet.go:493] "Attempting to sync node with API server" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509116 4992 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509126 4992 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509135 4992 kubelet.go:405] "Adding apiserver pod source" Nov 03 01:17:44.509326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.509144 4992 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 01:17:44.511033 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.511015 4992 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:17:44.511033 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.511033 4992 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:44.511703 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.511684 4992 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 01:17:44.529820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.529796 4992 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 01:17:44.529820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.529813 4992 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 01:17:44.529985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.529973 4992 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 01:17:44.530232 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530218 4992 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 01:17:44.530299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530255 4992 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 01:17:44.530387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530366 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 01:17:44.530387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530387 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 01:17:44.530387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530394 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530401 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530408 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530418 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530428 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530435 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530445 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530453 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530466 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530480 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530519 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 01:17:44.530604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.530528 4992 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 01:17:44.534681 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.534664 4992 server.go:1263] "Started kubelet" Nov 03 01:17:44.538017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.534863 4992 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 01:17:44.538017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.534878 4992 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 01:17:44.538017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.534964 4992 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 01:17:44.538017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.535273 4992 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 01:17:44.535206 ip-10-0-138-19 systemd[1]: Started Kubernetes Kubelet. Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.543967 4992 server.go:317] "Adding debug handlers to kubelet server" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.544421 4992 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.544592 4992 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.544605 4992 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.544662 4992 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.544670 4992 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.544878 4992 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.545592 4992 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:03 +0000 UTC" deadline="2028-05-29 23:35:30.33756698 +0000 UTC" Nov 03 01:17:44.545802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.545618 4992 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13774h17m45.791952685s" Nov 03 01:17:44.546356 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546100 4992 factory.go:55] Registering systemd factory Nov 03 01:17:44.546356 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546121 4992 factory.go:223] Registration of the systemd container factory successfully Nov 03 01:17:44.546721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546694 4992 factory.go:153] Registering CRI-O factory Nov 03 01:17:44.546721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546711 4992 factory.go:223] Registration of the crio container factory successfully Nov 03 01:17:44.546917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546783 4992 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 01:17:44.546917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546808 4992 factory.go:103] Registering Raw factory Nov 03 01:17:44.546917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.546826 4992 manager.go:1199] Started watching for new ooms in manager Nov 03 01:17:44.547194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.547180 4992 manager.go:319] Starting recovery of all containers Nov 03 01:17:44.548063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548001 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates" seLinuxMountContext="" Nov 03 01:17:44.548063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548042 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d87eeb-c7ba-4afa-bb30-9a512102f939" volumeName="kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy" seLinuxMountContext="" Nov 03 01:17:44.548063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548057 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35bfbf47-81e9-4f0c-818c-965de586692b" volumeName="kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548073 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502bb646-4048-4af0-90bb-a4f4688fe423" volumeName="kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548085 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548099 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548112 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4" volumeName="kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548125 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502bb646-4048-4af0-90bb-a4f4688fe423" volumeName="kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548140 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bdb27b8-f440-43c6-a0fa-152432024154" volumeName="kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548152 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e263b05-c71a-4d41-99d7-209ec4dbad6a" volumeName="kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548165 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91186c34-ad26-42d9-a4e4-70f9da7a10db" volumeName="kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548178 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954cc14b-f026-4cf9-93cb-b14e3b7ef933" volumeName="kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548190 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add38c69-7c29-44f2-940f-061783060dcd" volumeName="kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548205 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0657ec93-7ca2-415d-8ce6-d7a6be73d8fa" volumeName="kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548231 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477e52dd-4c41-4415-ada1-414b13751aef" volumeName="kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert" seLinuxMountContext="" Nov 03 01:17:44.548249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548248 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57f9e78d-c5d8-4af3-bbbb-15fddc658e70" volumeName="kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548264 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764c5ea1-9edc-4a86-8bec-b4001b6ad8c8" volumeName="kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548278 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="835f0a42-f920-4d77-9178-4a174fe526c1" volumeName="kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548290 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548302 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548314 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548326 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bdb27b8-f440-43c6-a0fa-152432024154" volumeName="kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548338 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548352 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbd9a86f-c50f-41b2-9c20-fe94fda357c1" volumeName="kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548373 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548389 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3deae577-9838-4896-82bf-a1b5b5710d3a" volumeName="kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548401 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5493a968-afc8-4a7d-aa18-22e2a14b8d70" volumeName="kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548415 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7d5877-a1e3-4c88-af99-bf4911871bbe" volumeName="kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548440 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954cc14b-f026-4cf9-93cb-b14e3b7ef933" volumeName="kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548475 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d87eeb-c7ba-4afa-bb30-9a512102f939" volumeName="kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548502 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d87eeb-c7ba-4afa-bb30-9a512102f939" volumeName="kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548515 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38df53d9-0193-43e6-a9a1-5ecd8fb2e456" volumeName="kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548541 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548568 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac23fba7-e4fe-45bc-8360-9875079da1cd" volumeName="kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548582 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548597 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38df53d9-0193-43e6-a9a1-5ecd8fb2e456" volumeName="kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548623 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec03c17-824a-4f8e-9a62-c1df47789ec9" volumeName="kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548659 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91186c34-ad26-42d9-a4e4-70f9da7a10db" volumeName="kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548683 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91186c34-ad26-42d9-a4e4-70f9da7a10db" volumeName="kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548696 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="080c75f8-1ec9-432b-aede-34b6d715ab6f" volumeName="kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548709 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4" volumeName="kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st" seLinuxMountContext="" Nov 03 01:17:44.548741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548733 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b7d5877-a1e3-4c88-af99-bf4911871bbe" volumeName="kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548746 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8895329f-6ed6-41c8-9f6b-d268dff1c4dc" volumeName="kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548772 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548795 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b871da5c-3f46-4189-ab3c-b8f94f9111d9" volumeName="kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548808 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35bfbf47-81e9-4f0c-818c-965de586692b" volumeName="kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548834 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548847 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43445eef-1fab-461d-a4db-bf26cb747247" volumeName="kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548869 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add38c69-7c29-44f2-940f-061783060dcd" volumeName="kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548884 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548899 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efde973b-4882-427d-a08a-b0b14ef8aaa0" volumeName="kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548923 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548935 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477e52dd-4c41-4415-ada1-414b13751aef" volumeName="kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548948 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="502bb646-4048-4af0-90bb-a4f4688fe423" volumeName="kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548971 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88af5e8b-13e7-47c8-9c9d-4c861f7c340b" volumeName="kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.548984 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549025 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549039 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9" volumeName="kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549052 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38ea1aba-6e28-4744-b0b2-af86439e36cb" volumeName="kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549066 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3deae577-9838-4896-82bf-a1b5b5710d3a" volumeName="kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549080 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3deae577-9838-4896-82bf-a1b5b5710d3a" volumeName="kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549092 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43445eef-1fab-461d-a4db-bf26cb747247" volumeName="kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549107 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764c5ea1-9edc-4a86-8bec-b4001b6ad8c8" volumeName="kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549121 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="835f0a42-f920-4d77-9178-4a174fe526c1" volumeName="kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549133 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8895329f-6ed6-41c8-9f6b-d268dff1c4dc" volumeName="kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549148 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cacd2582-2d59-4dc3-ba1f-28118cb27609" volumeName="kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549161 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8895329f-6ed6-41c8-9f6b-d268dff1c4dc" volumeName="kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549173 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549195 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efde973b-4882-427d-a08a-b0b14ef8aaa0" volumeName="kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549208 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39aa5aec-9458-4a7e-928a-7d00a4f84182" volumeName="kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549221 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3deae577-9838-4896-82bf-a1b5b5710d3a" volumeName="kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549238 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="477e52dd-4c41-4415-ada1-414b13751aef" volumeName="kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549252 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954cc14b-f026-4cf9-93cb-b14e3b7ef933" volumeName="kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549264 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" volumeName="kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549277 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efde973b-4882-427d-a08a-b0b14ef8aaa0" volumeName="kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549306 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc" volumeName="kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549318 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35bfbf47-81e9-4f0c-818c-965de586692b" volumeName="kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549332 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35bfbf47-81e9-4f0c-818c-965de586692b" volumeName="kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549367 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3deae577-9838-4896-82bf-a1b5b5710d3a" volumeName="kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549381 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="835f0a42-f920-4d77-9178-4a174fe526c1" volumeName="kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549394 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac23fba7-e4fe-45bc-8360-9875079da1cd" volumeName="kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549406 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954cc14b-f026-4cf9-93cb-b14e3b7ef933" volumeName="kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549418 4992 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efde973b-4882-427d-a08a-b0b14ef8aaa0" volumeName="kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle" seLinuxMountContext="" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549441 4992 reconstruct.go:97] "Volume reconstruction finished" Nov 03 01:17:44.549868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.549449 4992 reconciler.go:29] "Reconciler: start to sync state" Nov 03 01:17:44.561921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.561901 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/1.log" Nov 03 01:17:44.564020 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.563998 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/1.log" Nov 03 01:17:44.583723 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.583683 4992 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 01:17:44.586722 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.586702 4992 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 01:17:44.586813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.586727 4992 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 01:17:44.586813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.586747 4992 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 01:17:44.586813 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.586780 4992 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 01:17:44.614271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.614195 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/1.log" Nov 03 01:17:44.614271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.614229 4992 generic.go:356] "Generic (PLEG): container finished" podID="35bfbf47-81e9-4f0c-818c-965de586692b" containerID="711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228" exitCode=255 Nov 03 01:17:44.619644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619609 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="315b7228d65dcf7770087697e28ca3df45c819613912171d711d9bd5001dfda5" exitCode=0 Nov 03 01:17:44.619644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619641 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="69e5697e8228e693005d6b6bd29b7fc9e16c4eb0fca43732a955dbc3b1023f64" exitCode=0 Nov 03 01:17:44.619837 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619652 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="5a9c670559fd7484730ac35ecf63801128e455837cb657e5fe22489dc188ec99" exitCode=0 Nov 03 01:17:44.619837 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619660 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="316f0a31011564fbcdc99c26b4d96d8fd29b11145b158f535b9283a07b8be6b1" exitCode=0 Nov 03 01:17:44.619837 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619671 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="409cdad87b9ddf3ea5f4bcb84901b2cdf209514ac230ce6ac0326c025b18f0eb" exitCode=0 Nov 03 01:17:44.619837 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.619679 4992 generic.go:356] "Generic (PLEG): container finished" podID="954cc14b-f026-4cf9-93cb-b14e3b7ef933" containerID="aa6b89951836e6968e213095ca4aef2a621ef3d18f4897807770c53b2298f949" exitCode=0 Nov 03 01:17:44.623245 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.623222 4992 generic.go:356] "Generic (PLEG): container finished" podID="bb5781414dea338e9e0fd077df61848e" containerID="fafd1b7b774ac9d37676bffe32f8dde006ec22ae18aa05fb3ada306d9b035f41" exitCode=0 Nov 03 01:17:44.644698 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.644669 4992 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:44.647510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.647491 4992 manager.go:324] Recovery completed Nov 03 01:17:44.666123 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.666104 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.668498 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.668481 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.668604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.668513 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.668604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.668528 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.671153 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671136 4992 cpu_manager.go:226] "Starting" policy="none" Nov 03 01:17:44.671153 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671151 4992 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 01:17:44.671248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671153 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.671248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671172 4992 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 01:17:44.671310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671291 4992 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671302 4992 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671319 4992 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671325 4992 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671330 4992 policy_none.go:50] "Start" Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671338 4992 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 01:17:44.671360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671346 4992 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 01:17:44.671522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671467 4992 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 01:17:44.671522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671473 4992 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 01:17:44.671522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.671479 4992 policy_none.go:44] "Start" Nov 03 01:17:44.672022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.672005 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.672087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.672036 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.672087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.672047 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.675795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.675768 4992 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 01:17:44.678265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678250 4992 manager.go:341] "Starting Device Plugin manager" Nov 03 01:17:44.678352 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.678273 4992 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 01:17:44.678352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678282 4992 server.go:87] "Starting device plugin registration server" Nov 03 01:17:44.678551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678538 4992 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 01:17:44.678638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678550 4992 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 01:17:44.678687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678658 4992 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 01:17:44.678867 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678830 4992 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 01:17:44.678975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678886 4992 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 01:17:44.678975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678893 4992 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 01:17:44.679072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.678942 4992 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:17:44.678868551Z","UUID":"6fe84fbd-b80f-45ee-a373-b49951851478","Handler":null,"Name":"","Endpoint":""} Nov 03 01:17:44.680387 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.680365 4992 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 01:17:44.680456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.680396 4992 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 01:17:44.680456 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.680403 4992 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" Nov 03 01:17:44.680456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.680428 4992 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 01:17:44.686889 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.686855 4992 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal"] Nov 03 01:17:44.686981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.686970 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.687709 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.687694 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.687800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.687723 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.687800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.687739 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.688079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688065 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.688159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688141 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.688756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688740 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.688833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688764 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.688833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688777 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.689046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688834 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.689046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688860 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.689046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.688872 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.689046 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.688901 4992 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.689222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689191 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"b5ac7307a30e6ec216df9c122c45451f","Type":"ContainerStarted","Data":"526794604f5c99621cb9ba8aaeb77563646274f6d7aba9bc507887eb3a8ca17b"} Nov 03 01:17:44.689263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689234 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"b5ac7307a30e6ec216df9c122c45451f","Type":"ContainerStarted","Data":"f209e36f09f920ff95779de234e3aaf08c5dc54ceccd5f37aed4f2b8142f6d60"} Nov 03 01:17:44.689306 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689289 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.689356 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689288 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerStarted","Data":"64e6cd9aa3a0d259669215b337e578fa8b97da3d0c400068ec070dea0ca662ef"} Nov 03 01:17:44.689356 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689335 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerDied","Data":"fafd1b7b774ac9d37676bffe32f8dde006ec22ae18aa05fb3ada306d9b035f41"} Nov 03 01:17:44.689450 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689355 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" event={"ID":"bb5781414dea338e9e0fd077df61848e","Type":"ContainerStarted","Data":"4e49327317a44fd238415b877961f1a37e251cb80f42e6534707846bcd23c790"} Nov 03 01:17:44.689983 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689966 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.690044 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.689998 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.690044 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.690012 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.690153 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.690140 4992 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.us-west-2.compute.internal\" not found" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.779047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.779016 4992 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 01:17:44.780082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.780057 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 01:17:44.780189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.780118 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 01:17:44.780189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.780134 4992 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-19.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 01:17:44.780189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.780154 4992 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.782474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.782456 4992 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 01:17:44.782844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.782827 4992 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:44.782916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.782856 4992 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:44.784700 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.784678 4992 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:44.784917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.784904 4992 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 01:17:44.846194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.846160 4992 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.883455 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.883346 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.883455 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.883430 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.883455 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.883451 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.892932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.892913 4992 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.893015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.892982 4992 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.909397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.909376 4992 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:44.909528 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:44.909418 4992 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.909528 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.909433 4992 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984306 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984276 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984306 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984308 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984442 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984326 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984442 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984385 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984442 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984439 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb5781414dea338e9e0fd077df61848e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"bb5781414dea338e9e0fd077df61848e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:44.984552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:44.984469 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b5ac7307a30e6ec216df9c122c45451f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" (UID: \"b5ac7307a30e6ec216df9c122c45451f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.016888 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.016865 4992 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:45.016993 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.016906 4992 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.509868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.509827 4992 apiserver.go:52] "Watching apiserver" Nov 03 01:17:45.649059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.649032 4992 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.649203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.649155 4992 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.669325 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.669300 4992 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 01:17:45.670091 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.670062 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm","openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp","openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg","openshift-dns/dns-default-ms9cn","openshift-network-diagnostics/network-check-target-qk55d","kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal","openshift-image-registry/image-registry-769d849c4c-bmjrw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-6pm5n","openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz","openshift-dns/node-resolver-rv9vt","openshift-network-console/networking-console-plugin-86c478bb88-4frnh","kube-system/global-pull-secret-syncer-xqklw","openshift-ingress/router-default-85ff878dc9-x2zvj","openshift-service-ca/service-ca-5fffc97bdf-clq7z","kube-system/kas-connection-checker-685895c7c9-5kgrh","openshift-multus/multus-additional-cni-plugins-6g9zq","kube-system/konnectivity-agent-tjn7g","openshift-image-registry/node-ca-jkcgz","openshift-ovn-kubernetes/ovnkube-node-8k5j6","openshift-ingress-canary/ingress-canary-rrblm","openshift-multus/network-metrics-daemon-x9lj4","openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6","openshift-network-operator/iptables-alerter-9wq78","openshift-image-registry/image-registry-c45667b98-7dq66","openshift-multus/multus-vfccq","openshift-insights/insights-runtime-extractor-ktk4w","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln","kube-system/kas-connection-checker-685895c7c9-7rq8q","openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9","openshift-console-operator/console-operator-7cd56b8dbc-6jftg","openshift-insights/periodic-gathering-sdbcx-x69tk","openshift-insights/insights-operator-7c4986d449-tvs7q","openshift-cluster-node-tuning-operator/tuned-d64cz"] Nov 03 01:17:45.672260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.672239 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.673056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.673039 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.673501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.673479 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.674812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.674792 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.675307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.675291 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:45.681251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.681229 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:45.683716 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.683696 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.684220 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.684204 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.686247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.686228 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.690649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.690630 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.712189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.712170 4992 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:45.712270 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.712214 4992 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.712531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.712516 4992 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 01:17:45.712583 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.712567 4992 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.us-west-2.compute.internal" Nov 03 01:17:45.728495 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728471 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728496 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728504 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728531 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728577 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728586 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728590 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 01:17:45.728609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728594 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 01:17:45.728834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.728632 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 01:17:45.729099 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729083 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729253 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729305 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729290 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729342 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729419 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729403 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729622 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729644 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 01:17:45.729721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729660 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729682 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729674 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729696 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729723 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729731 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729737 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729733 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729755 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729732 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729766 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729818 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.729895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729893 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 01:17:45.730172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729929 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.730172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729938 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 01:17:45.730172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.729969 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.730366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.730348 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.747046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.747023 4992 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 01:17:45.765313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.765252 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 01:17:45.782407 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.782377 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\"" Nov 03 01:17:45.782602 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.782531 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh6sf\"" Nov 03 01:17:45.782920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.782630 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\"" Nov 03 01:17:45.783191 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783173 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-8lzdq\"" Nov 03 01:17:45.783396 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783375 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 01:17:45.783616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783595 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.783679 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783638 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.783679 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783662 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 01:17:45.783781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783765 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 01:17:45.783985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783968 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 01:17:45.784046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.783998 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\"" Nov 03 01:17:45.784448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.784431 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 01:17:45.785625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.785605 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.785720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.785659 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.785975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.785960 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786021 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 01:17:45.786096 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786084 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 01:17:45.786230 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786211 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 01:17:45.786378 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786363 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.786611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786593 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 01:17:45.786666 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786616 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.786666 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786644 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786754 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786717 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786755 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786805 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 01:17:45.786857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786817 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 01:17:45.786857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786843 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.786857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786854 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786805 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786880 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786904 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786928 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786884 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 01:17:45.786948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786912 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786978 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.786991 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787006 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787034 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787063 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787070 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787116 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\"" Nov 03 01:17:45.787150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787132 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\"" Nov 03 01:17:45.787389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787195 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\"" Nov 03 01:17:45.787389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787237 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 01:17:45.787389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787346 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.787389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787363 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 01:17:45.787750 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787735 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 01:17:45.787792 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787753 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.787829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787817 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 01:17:45.787933 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787922 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\"" Nov 03 01:17:45.787964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787944 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 01:17:45.788000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.787972 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.788055 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788041 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 01:17:45.788090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788061 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 01:17:45.788118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788091 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 01:17:45.788176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788162 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 01:17:45.788176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788171 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\"" Nov 03 01:17:45.788237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788198 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-27r6g\"" Nov 03 01:17:45.788237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788225 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 01:17:45.788298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788270 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\"" Nov 03 01:17:45.788298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788290 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\"" Nov 03 01:17:45.788353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788332 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 01:17:45.788498 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788486 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 01:17:45.788540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788509 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wq28x\"" Nov 03 01:17:45.788595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788543 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.788651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788596 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t67t\" (UniqueName: \"kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.788651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788571 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\"" Nov 03 01:17:45.788651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788549 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\"" Nov 03 01:17:45.788774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788664 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.788821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788769 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\"" Nov 03 01:17:45.788821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788807 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 01:17:45.788917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788837 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 01:17:45.788917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788850 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:45.788917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788879 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.788917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788888 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\"" Nov 03 01:17:45.788917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788855 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788905 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788928 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788957 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhc7\" (UniqueName: \"kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788984 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502bb646-4048-4af0-90bb-a4f4688fe423-tmp-dir\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788993 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.788934 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vdn44\"" Nov 03 01:17:45.789087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789061 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789091 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld79v\" (UniqueName: \"kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789118 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffq6\" (UniqueName: \"kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789144 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kbl\" (UniqueName: \"kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789175 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh5kw\" (UniqueName: \"kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw\") pod \"migrator-66df5d496b-bssmz\" (UID: \"5493a968-afc8-4a7d-aa18-22e2a14b8d70\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789254 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/add38c69-7c29-44f2-940f-061783060dcd-agent-certs\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789379 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/add38c69-7c29-44f2-940f-061783060dcd-konnectivity-ca\") pod \"konnectivity-agent-tjn7g\" (UID: \"add38c69-7c29-44f2-940f-061783060dcd\") " pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789400 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789434 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:45.789486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789480 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789509 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789533 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5j5g\" (UniqueName: \"kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789597 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknc5\" (UniqueName: \"kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5\") pod \"network-check-source-6f78d59f99-9rfn6\" (UID: \"080c75f8-1ec9-432b-aede-34b6d715ab6f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789646 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789603 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789678 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbb8\" (UniqueName: \"kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789704 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789729 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789756 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789782 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789820 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789851 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqp2\" (UniqueName: \"kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2\") pod \"volume-data-source-validator-58c9f5f46d-jldp9\" (UID: \"b871da5c-3f46-4189-ab3c-b8f94f9111d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789877 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789873 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-trusted-ca\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789911 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789937 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-key\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.789980 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.790002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790001 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790032 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790095 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790126 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790134 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790171 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790186 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790203 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790220 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790254 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790286 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb69j\" (UniqueName: \"kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790351 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/502bb646-4048-4af0-90bb-a4f4688fe423-config-volume\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790378 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790388 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790420 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790464 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790486 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790509 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790591 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-original-pull-secret\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790588 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790695 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-cabundle\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790702 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-default-certificate\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790724 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790767 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4st\" (UniqueName: \"kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790775 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.790795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790811 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790859 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790899 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790905 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790925 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.790979 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791004 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791016 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cgr\" (UniqueName: \"kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791048 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cs85\" (UniqueName: \"kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85\") pod \"kas-connection-checker-685895c7c9-5kgrh\" (UID: \"57f9e78d-c5d8-4af3-bbbb-15fddc658e70\") " pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791067 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791082 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791097 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791100 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791114 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bxt\" (UniqueName: \"kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791174 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-tmp\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791208 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791230 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791259 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791284 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-script-lib\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791293 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791302 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35bfbf47-81e9-4f0c-818c-965de586692b-serving-cert\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791383 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791425 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791427 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cacd2582-2d59-4dc3-ba1f-28118cb27609-service-ca-bundle\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791499 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791517 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791545 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791551 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-daemon-config\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791585 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791659 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43445eef-1fab-461d-a4db-bf26cb747247-serviceca\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791697 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.791752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791733 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vsq\" (UniqueName: \"kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791882 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791937 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791974 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.791997 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792041 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792095 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792186 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792241 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792249 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-binary-copy\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792271 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792308 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792360 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792411 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792442 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792465 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-ovnkube-config\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792499 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65hc\" (UniqueName: \"kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792541 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792550 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35bfbf47-81e9-4f0c-818c-965de586692b-config\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792589 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792613 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792630 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792649 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792685 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792753 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-tuned\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792761 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792768 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3deae577-9838-4896-82bf-a1b5b5710d3a-ovn-node-metrics-cert\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792766 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-tmp\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792790 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792829 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792856 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792885 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.792905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792909 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cacd2582-2d59-4dc3-ba1f-28118cb27609-serving-cert\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792932 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792936 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.792979 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793007 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793047 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793065 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793084 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwj4\" (UniqueName: \"kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793049 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ec03c17-824a-4f8e-9a62-c1df47789ec9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793100 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwq4\" (UniqueName: \"kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4\") pod \"kas-connection-checker-685895c7c9-6pm5n\" (UID: \"88af5e8b-13e7-47c8-9c9d-4c861f7c340b\") " pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793166 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793200 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7d5877-a1e3-4c88-af99-bf4911871bbe-cert\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793218 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793261 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793295 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793321 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793370 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793398 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793406 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-data-volume\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793438 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6qg\" (UniqueName: \"kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg\") pod \"migrator-66df5d496b-8vdx2\" (UID: \"38ea1aba-6e28-4744-b0b2-af86439e36cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793474 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-259tj\" (UniqueName: \"kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793502 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793506 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793527 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793535 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793553 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793606 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793632 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793663 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793683 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3deae577-9838-4896-82bf-a1b5b5710d3a-env-overrides\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793699 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793741 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6j4\" (UniqueName: \"kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793801 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6z65\" (UniqueName: \"kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793869 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793894 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793923 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793951 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.793970 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794009 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794046 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac23fba7-e4fe-45bc-8360-9875079da1cd-tmp-dir\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794082 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794111 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794136 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794162 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31d87eeb-c7ba-4afa-bb30-9a512102f939-cni-binary-copy\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794240 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794245 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2m9b\" (UniqueName: \"kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.794295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794266 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cacd2582-2d59-4dc3-ba1f-28118cb27609-snapshots\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794358 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794371 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-iptables-alerter-script\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794410 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794412 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/954cc14b-f026-4cf9-93cb-b14e3b7ef933-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794483 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794503 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst7s\" (UniqueName: \"kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s\") pod \"kas-connection-checker-685895c7c9-7rq8q\" (UID: \"8e263b05-c71a-4d41-99d7-209ec4dbad6a\") " pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794520 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794542 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794520 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477e52dd-4c41-4415-ada1-414b13751aef-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794585 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794595 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794625 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlnp\" (UniqueName: \"kubernetes.io/projected/8a2fff98-9f33-4535-b4e8-9594fd287370-kube-api-access-qrlnp\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794645 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794687 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794719 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794752 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794789 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794810 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-serving-cert\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794833 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794868 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794935 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794965 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.794989 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795077 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795088 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-config\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795105 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvpm\" (UniqueName: \"kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795131 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795157 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795286 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-stats-auth\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.795774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.795327 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477e52dd-4c41-4415-ada1-414b13751aef-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.862641 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.862609 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.867454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.867435 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8qt\" (UniqueName: \"kubernetes.io/projected/e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9-kube-api-access-xn8qt\") pod \"network-check-target-qk55d\" (UID: \"e7d3daaa-f3ab-4d3f-97e8-71ac93ee40a9\") " pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:45.868725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.868706 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t67t\" (UniqueName: \"kubernetes.io/projected/3deae577-9838-4896-82bf-a1b5b5710d3a-kube-api-access-2t67t\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.873358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.873332 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.873948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.873688 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.873948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.873793 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.873948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.873876 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cgr\" (UniqueName: \"kubernetes.io/projected/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-api-access-j2cgr\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.874078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.874001 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvpm\" (UniqueName: \"kubernetes.io/projected/835f0a42-f920-4d77-9178-4a174fe526c1-kube-api-access-wkvpm\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.874078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.874062 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cs85\" (UniqueName: \"kubernetes.io/projected/57f9e78d-c5d8-4af3-bbbb-15fddc658e70-kube-api-access-2cs85\") pod \"kas-connection-checker-685895c7c9-5kgrh\" (UID: \"57f9e78d-c5d8-4af3-bbbb-15fddc658e70\") " pod="kube-system/kas-connection-checker-685895c7c9-5kgrh" Nov 03 01:17:45.874389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.874370 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65hc\" (UniqueName: \"kubernetes.io/projected/954cc14b-f026-4cf9-93cb-b14e3b7ef933-kube-api-access-r65hc\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.874420 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.874370 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffq6\" (UniqueName: \"kubernetes.io/projected/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-kube-api-access-mffq6\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.878129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.878108 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld79v\" (UniqueName: \"kubernetes.io/projected/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-kube-api-access-ld79v\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:45.878304 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.878289 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5j5g\" (UniqueName: \"kubernetes.io/projected/91186c34-ad26-42d9-a4e4-70f9da7a10db-kube-api-access-p5j5g\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.883782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.883753 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6qg\" (UniqueName: \"kubernetes.io/projected/38ea1aba-6e28-4744-b0b2-af86439e36cb-kube-api-access-jv6qg\") pod \"migrator-66df5d496b-8vdx2\" (UID: \"38ea1aba-6e28-4744-b0b2-af86439e36cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" Nov 03 01:17:45.884222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884147 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb69j\" (UniqueName: \"kubernetes.io/projected/477e52dd-4c41-4415-ada1-414b13751aef-kube-api-access-sb69j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln\" (UID: \"477e52dd-4c41-4415-ada1-414b13751aef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" Nov 03 01:17:45.884222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884205 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6j4\" (UniqueName: \"kubernetes.io/projected/43445eef-1fab-461d-a4db-bf26cb747247-kube-api-access-tg6j4\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.884390 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884345 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"periodic-gathering-sdbcx-x69tk\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:17:45.884678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884416 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknc5\" (UniqueName: \"kubernetes.io/projected/080c75f8-1ec9-432b-aede-34b6d715ab6f-kube-api-access-wknc5\") pod \"network-check-source-6f78d59f99-9rfn6\" (UID: \"080c75f8-1ec9-432b-aede-34b6d715ab6f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-9rfn6" Nov 03 01:17:45.884678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884473 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwj4\" (UniqueName: \"kubernetes.io/projected/8895329f-6ed6-41c8-9f6b-d268dff1c4dc-kube-api-access-xxwj4\") pod \"service-ca-operator-544d749c8c-rbtvp\" (UID: \"8895329f-6ed6-41c8-9f6b-d268dff1c4dc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" Nov 03 01:17:45.884678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884575 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhc7\" (UniqueName: \"kubernetes.io/projected/ac23fba7-e4fe-45bc-8360-9875079da1cd-kube-api-access-2dhc7\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.884920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884678 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2m9b\" (UniqueName: \"kubernetes.io/projected/cacd2582-2d59-4dc3-ba1f-28118cb27609-kube-api-access-n2m9b\") pod \"insights-operator-7c4986d449-tvs7q\" (UID: \"cacd2582-2d59-4dc3-ba1f-28118cb27609\") " pod="openshift-insights/insights-operator-7c4986d449-tvs7q" Nov 03 01:17:45.884920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884703 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwq4\" (UniqueName: \"kubernetes.io/projected/88af5e8b-13e7-47c8-9c9d-4c861f7c340b-kube-api-access-2zwq4\") pod \"kas-connection-checker-685895c7c9-6pm5n\" (UID: \"88af5e8b-13e7-47c8-9c9d-4c861f7c340b\") " pod="kube-system/kas-connection-checker-685895c7c9-6pm5n" Nov 03 01:17:45.885065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.884988 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6z65\" (UniqueName: \"kubernetes.io/projected/6b7d5877-a1e3-4c88-af99-bf4911871bbe-kube-api-access-v6z65\") pod \"ingress-canary-rrblm\" (UID: \"6b7d5877-a1e3-4c88-af99-bf4911871bbe\") " pod="openshift-ingress-canary/ingress-canary-rrblm" Nov 03 01:17:45.885065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885007 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst7s\" (UniqueName: \"kubernetes.io/projected/8e263b05-c71a-4d41-99d7-209ec4dbad6a-kube-api-access-lst7s\") pod \"kas-connection-checker-685895c7c9-7rq8q\" (UID: \"8e263b05-c71a-4d41-99d7-209ec4dbad6a\") " pod="kube-system/kas-connection-checker-685895c7c9-7rq8q" Nov 03 01:17:45.885065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885017 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-259tj\" (UniqueName: \"kubernetes.io/projected/502bb646-4048-4af0-90bb-a4f4688fe423-kube-api-access-259tj\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.885228 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885150 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bxt\" (UniqueName: \"kubernetes.io/projected/35bfbf47-81e9-4f0c-818c-965de586692b-kube-api-access-26bxt\") pod \"console-operator-7cd56b8dbc-6jftg\" (UID: \"35bfbf47-81e9-4f0c-818c-965de586692b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:45.885228 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885182 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kbl\" (UniqueName: \"kubernetes.io/projected/31d87eeb-c7ba-4afa-bb30-9a512102f939-kube-api-access-s2kbl\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.885335 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885246 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4st\" (UniqueName: \"kubernetes.io/projected/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-kube-api-access-6f4st\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.885399 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885384 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vsq\" (UniqueName: \"kubernetes.io/projected/5bdb27b8-f440-43c6-a0fa-152432024154-kube-api-access-47vsq\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.885542 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885523 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqp2\" (UniqueName: \"kubernetes.io/projected/b871da5c-3f46-4189-ab3c-b8f94f9111d9-kube-api-access-qgqp2\") pod \"volume-data-source-validator-58c9f5f46d-jldp9\" (UID: \"b871da5c-3f46-4189-ab3c-b8f94f9111d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-jldp9" Nov 03 01:17:45.885774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885754 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh5kw\" (UniqueName: \"kubernetes.io/projected/5493a968-afc8-4a7d-aa18-22e2a14b8d70-kube-api-access-qh5kw\") pod \"migrator-66df5d496b-bssmz\" (UID: \"5493a968-afc8-4a7d-aa18-22e2a14b8d70\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" Nov 03 01:17:45.885817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.885795 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbb8\" (UniqueName: \"kubernetes.io/projected/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kube-api-access-vhbb8\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.895498 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895481 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.895603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895505 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.895603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895521 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.895603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895538 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:45.895603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895569 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895603 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895611 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-cnibin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895620 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895638 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895642 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895670 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895673 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-dbus\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895677 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-cnibin\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.895684 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.895701 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895675 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-multus-certs\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895717 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895725 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-registration-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895688 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-ovn\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.895752 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.395735603 +0000 UTC m=+1.974558172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:45.895781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895780 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895797 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895818 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895844 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895851 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-netd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895868 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-crio-socket\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895870 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895897 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895902 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-k8s-cni-cncf-io\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.895908 4992 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895920 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.895961 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.395949589 +0000 UTC m=+1.974772172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895898 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-bin\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.895987 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896003 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896018 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896034 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896049 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896064 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896080 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896081 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-hostroot\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896087 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-system-cni-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896095 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43445eef-1fab-461d-a4db-bf26cb747247-host\") pod \"node-ca-jkcgz\" (UID: \"43445eef-1fab-461d-a4db-bf26cb747247\") " pod="openshift-image-registry/node-ca-jkcgz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896095 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896124 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-device-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896134 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896157 4992 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896163 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896192 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896199 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-run-netns\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896211 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396195705 +0000 UTC m=+1.975018284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : secret "metrics-daemon-secret" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896159 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysconfig\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896221 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896246 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896257 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896276 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896291 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896308 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896322 4992 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896328 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896248 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764c5ea1-9edc-4a86-8bec-b4001b6ad8c8-host-slash\") pod \"iptables-alerter-9wq78\" (UID: \"764c5ea1-9edc-4a86-8bec-b4001b6ad8c8\") " pod="openshift-network-operator/iptables-alerter-9wq78" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896358 4992 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896374 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396360081 +0000 UTC m=+1.975182660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896354 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896394 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-slash\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896413 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396398409 +0000 UTC m=+1.975221006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896424 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cbd9a86f-c50f-41b2-9c20-fe94fda357c1-kubelet-config\") pod \"global-pull-secret-syncer-xqklw\" (UID: \"cbd9a86f-c50f-41b2-9c20-fe94fda357c1\") " pod="kube-system/global-pull-secret-syncer-xqklw" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896431 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396421811 +0000 UTC m=+1.975244406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896458 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.896434 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896430 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896484 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896483 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896507 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896528 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396516857 +0000 UTC m=+1.975339439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896530 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-systemd\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896547 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896574 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-sys\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896547 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/954cc14b-f026-4cf9-93cb-b14e3b7ef933-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6g9zq\" (UID: \"954cc14b-f026-4cf9-93cb-b14e3b7ef933\") " pod="openshift-multus/multus-additional-cni-plugins-6g9zq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896591 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlnp\" (UniqueName: \"kubernetes.io/projected/8a2fff98-9f33-4535-b4e8-9594fd287370-kube-api-access-qrlnp\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896620 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896623 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896643 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896592 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-kubelet\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896667 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896675 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-socket-dir\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896688 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896700 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-sys-fs\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896709 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896718 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-modprobe-d\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896736 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896763 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896801 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-sysctl-conf\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896807 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-log-socket\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896805 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896811 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-cni-multus\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896846 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896869 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896846 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-host\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896888 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896846 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-etc-kubernetes\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896901 4992 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896905 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896926 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896923 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-etc-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896941 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0657ec93-7ca2-415d-8ce6-d7a6be73d8fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-rdcm6\" (UID: \"0657ec93-7ca2-415d-8ce6-d7a6be73d8fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rdcm6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896947 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-systemd-units\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.896934 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.396925101 +0000 UTC m=+1.975747670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896965 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-var-lib-kubelet\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896983 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.896998 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897017 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897041 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-key\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897052 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-run-systemd\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897061 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-host-var-lib-kubelet\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897065 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.897074 4992 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897091 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.897113 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.397099962 +0000 UTC m=+1.975922555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897137 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897149 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-system-cni-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897136 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-socket-dir-parent\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897167 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-multus-conf-dir\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897164 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897201 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac23fba7-e4fe-45bc-8360-9875079da1cd-hosts-file\") pod \"node-resolver-rv9vt\" (UID: \"ac23fba7-e4fe-45bc-8360-9875079da1cd\") " pod="openshift-dns/node-resolver-rv9vt" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897205 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897227 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31d87eeb-c7ba-4afa-bb30-9a512102f939-etc-kubernetes\") pod \"multus-vfccq\" (UID: \"31d87eeb-c7ba-4afa-bb30-9a512102f939\") " pod="openshift-multus/multus-vfccq" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897232 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897257 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897283 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897296 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-lib-modules\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897307 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-cabundle\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897319 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-var-lib-openvswitch\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897332 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897354 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897323 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-node-log\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897355 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-netns\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897329 4992 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897372 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897390 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897411 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897445 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.897456 4992 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897465 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3deae577-9838-4896-82bf-a1b5b5710d3a-host-cni-bin\") pod \"ovnkube-node-8k5j6\" (UID: \"3deae577-9838-4896-82bf-a1b5b5710d3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897483 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91186c34-ad26-42d9-a4e4-70f9da7a10db-run\") pod \"tuned-d64cz\" (UID: \"91186c34-ad26-42d9-a4e4-70f9da7a10db\") " pod="openshift-cluster-node-tuning-operator/tuned-d64cz" Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:45.897507 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:46.397494013 +0000 UTC m=+1.976316593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:45.897980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.897846 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-cabundle\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.900791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.899385 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a2fff98-9f33-4535-b4e8-9594fd287370-signing-key\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.959888 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.959856 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlnp\" (UniqueName: \"kubernetes.io/projected/8a2fff98-9f33-4535-b4e8-9594fd287370-kube-api-access-qrlnp\") pod \"service-ca-5fffc97bdf-clq7z\" (UID: \"8a2fff98-9f33-4535-b4e8-9594fd287370\") " pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:45.971845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.971813 4992 scope.go:122] "RemoveContainer" containerID="711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228" Nov 03 01:17:45.987011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:45.986992 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:17:46.001967 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.001936 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" Nov 03 01:17:46.311191 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.311117 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rv9vt_ac23fba7-e4fe-45bc-8360-9875079da1cd/dns-node-resolver/0.log" Nov 03 01:17:46.369649 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:46.369598 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2fff98_9f33_4535_b4e8_9594fd287370.slice/crio-14448844bde200fb04dc229bd635d5d1579f74773711784296fbc5cc0b4da9a2 WatchSource:0}: Error finding container 14448844bde200fb04dc229bd635d5d1579f74773711784296fbc5cc0b4da9a2: Status 404 returned error can't find the container with id 14448844bde200fb04dc229bd635d5d1579f74773711784296fbc5cc0b4da9a2 Nov 03 01:17:46.370288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.370269 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-clq7z"] Nov 03 01:17:46.402492 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402464 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:46.402604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402500 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:46.402678 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402615 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:46.402678 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402626 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:46.402678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402626 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:46.402678 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402665 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.402644081 +0000 UTC m=+2.981466667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402717 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.402687664 +0000 UTC m=+2.981510237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402725 4992 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402770 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402780 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.402763958 +0000 UTC m=+2.981586538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402838 4992 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:46.402865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402850 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402877 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.402866166 +0000 UTC m=+2.981688774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402917 4992 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402956 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.402944193 +0000 UTC m=+2.981766771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.402916 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402977 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.402988 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.403012 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403020 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.40300916 +0000 UTC m=+2.981831729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.403059 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403069 4992 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.403103 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403109 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.403098934 +0000 UTC m=+2.981921535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.403136 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403144 4992 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 01:17:46.403162 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403161 4992 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:46.403699 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403193 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.403178817 +0000 UTC m=+2.982001400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : secret "metrics-daemon-secret" not found Nov 03 01:17:46.403699 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403211 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.403203511 +0000 UTC m=+2.982026078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:46.403699 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403237 4992 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:46.403699 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.403278 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:47.403266136 +0000 UTC m=+2.982088722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:46.653389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.653311 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:17:46.653840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.653800 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/1.log" Nov 03 01:17:46.653922 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.653837 4992 generic.go:356] "Generic (PLEG): container finished" podID="35bfbf47-81e9-4f0c-818c-965de586692b" containerID="fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b" exitCode=255 Nov 03 01:17:46.653971 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.653913 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" event={"ID":"35bfbf47-81e9-4f0c-818c-965de586692b","Type":"ContainerDied","Data":"fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b"} Nov 03 01:17:46.653971 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.653968 4992 scope.go:122] "RemoveContainer" containerID="711fdd62842df3b81c3fde47d39515c134c776e327ca314759a99de0b7611228" Nov 03 01:17:46.654318 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.654299 4992 scope.go:122] "RemoveContainer" containerID="fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b" Nov 03 01:17:46.654550 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:46.654531 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jftg_openshift-console-operator(35bfbf47-81e9-4f0c-818c-965de586692b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" podUID="35bfbf47-81e9-4f0c-818c-965de586692b" Nov 03 01:17:46.655595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.655547 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" event={"ID":"8a2fff98-9f33-4535-b4e8-9594fd287370","Type":"ContainerStarted","Data":"7eeb53b979b9e104adcae929992ba7b536f524187462a95937e4e69ab1fc9867"} Nov 03 01:17:46.655695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:46.655620 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" event={"ID":"8a2fff98-9f33-4535-b4e8-9594fd287370","Type":"ContainerStarted","Data":"14448844bde200fb04dc229bd635d5d1579f74773711784296fbc5cc0b4da9a2"} Nov 03 01:17:47.113295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.113265 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkcgz_43445eef-1fab-461d-a4db-bf26cb747247/node-ca/0.log" Nov 03 01:17:47.349626 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.349595 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:17:47.376209 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.376179 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413532 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413692 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.413710 4992 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413744 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.413781 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.413755664 +0000 UTC m=+4.992578244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : secret "metrics-daemon-secret" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413826 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.413850 4992 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413866 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.413917 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.413930 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.413914499 +0000 UTC m=+4.992737084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.413992 4992 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414035 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414023664 +0000 UTC m=+4.992846244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : secret "router-metrics-certs-default" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414051 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414041541 +0000 UTC m=+4.992864122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414109 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.414109 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414121 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414160 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414148709 +0000 UTC m=+4.992971274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.414187 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414217 4992 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.414234 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414258 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls podName:38df53d9-0193-43e6-a9a1-5ecd8fb2e456 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414244105 +0000 UTC m=+4.993066684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ktk4w" (UID: "38df53d9-0193-43e6-a9a1-5ecd8fb2e456") : secret "insights-runtime-extractor-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414298 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414308 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414345 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414334696 +0000 UTC m=+4.993157284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.414303 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414360 4992 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414399 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414387508 +0000 UTC m=+4.993210088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414857 4992 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414906 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls podName:3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414886427 +0000 UTC m=+4.993709008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-cs7fg" (UID: "3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4") : secret "cluster-monitoring-operator-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414953 4992 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 01:17:47.415591 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.414988 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls podName:502bb646-4048-4af0-90bb-a4f4688fe423 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:49.414972571 +0000 UTC m=+4.993795139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls") pod "dns-default-ms9cn" (UID: "502bb646-4048-4af0-90bb-a4f4688fe423") : secret "dns-default-metrics-tls" not found Nov 03 01:17:47.667622 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.662685 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8vdx2" event={"ID":"38ea1aba-6e28-4744-b0b2-af86439e36cb","Type":"ContainerStarted","Data":"981341e6187d1d9c10e4880c512b3eb4571aa481d50a8f197de0f5ff23517101"} Nov 03 01:17:47.669452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.669425 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:17:47.669895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.669876 4992 scope.go:122] "RemoveContainer" containerID="fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b" Nov 03 01:17:47.670120 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:47.670091 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jftg_openshift-console-operator(35bfbf47-81e9-4f0c-818c-965de586692b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" podUID="35bfbf47-81e9-4f0c-818c-965de586692b" Nov 03 01:17:47.671773 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.671750 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bssmz" event={"ID":"5493a968-afc8-4a7d-aa18-22e2a14b8d70","Type":"ContainerStarted","Data":"67400a320de29d77042345b1113e69ed38dfa5843292597ded488b2cf59912a9"} Nov 03 01:17:47.710720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.710699 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrblm_6b7d5877-a1e3-4c88-af99-bf4911871bbe/serve-healthcheck-canary/0.log" Nov 03 01:17:47.911450 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.911425 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8vdx2_38ea1aba-6e28-4744-b0b2-af86439e36cb/migrator/0.log" Nov 03 01:17:47.913609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:47.913584 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tjn7g" Nov 03 01:17:48.112122 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:48.112052 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-bssmz_5493a968-afc8-4a7d-aa18-22e2a14b8d70/migrator/0.log" Nov 03 01:17:48.313743 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:48.313714 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln_477e52dd-4c41-4415-ada1-414b13751aef/kube-storage-version-migrator-operator/0.log" Nov 03 01:17:48.321122 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:48.321073 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-clq7z" podStartSLOduration=3.32106195 podStartE2EDuration="3.32106195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:48.32057986 +0000 UTC m=+3.899402441" watchObservedRunningTime="2026-11-03 01:17:48.32106195 +0000 UTC m=+3.899884533" Nov 03 01:17:48.603413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:48.603389 4992 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:48.675647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:48.675620 4992 scope.go:122] "RemoveContainer" containerID="fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b" Nov 03 01:17:48.676056 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:48.675924 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6jftg_openshift-console-operator(35bfbf47-81e9-4f0c-818c-965de586692b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" podUID="35bfbf47-81e9-4f0c-818c-965de586692b" Nov 03 01:17:49.434660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434623 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434674 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434703 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434733 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434761 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434792 4992 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 01:17:49.434810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434806 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434845 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434869 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.434850441 +0000 UTC m=+9.013673029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : secret "metrics-daemon-secret" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434793 4992 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434898 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle podName:835f0a42-f920-4d77-9178-4a174fe526c1 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.434888647 +0000 UTC m=+9.013711215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle") pod "router-default-85ff878dc9-x2zvj" (UID: "835f0a42-f920-4d77-9178-4a174fe526c1") : configmap references non-existent config key: service-ca.crt Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434951 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.434954 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434965 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.434944869 +0000 UTC m=+9.013767448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.434969 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435021 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.435008832 +0000 UTC m=+9.013831414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.435017 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435079 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:49.435101 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435091 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:49.435644 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435123 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.435111821 +0000 UTC m=+9.013934391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:49.435644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.435077 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:49.435644 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435133 4992 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:49.435644 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:49.435172 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:53.435163197 +0000 UTC m=+9.013985776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:49.437090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.437068 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/502bb646-4048-4af0-90bb-a4f4688fe423-metrics-tls\") pod \"dns-default-ms9cn\" (UID: \"502bb646-4048-4af0-90bb-a4f4688fe423\") " pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:49.437212 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.437180 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-cs7fg\" (UID: \"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:49.437472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.437456 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835f0a42-f920-4d77-9178-4a174fe526c1-metrics-certs\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:49.437540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.437519 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38df53d9-0193-43e6-a9a1-5ecd8fb2e456-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ktk4w\" (UID: \"38df53d9-0193-43e6-a9a1-5ecd8fb2e456\") " pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:49.574003 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.573970 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" Nov 03 01:17:49.584487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.584459 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:49.586646 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.586620 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ktk4w" Nov 03 01:17:49.971429 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:49.971395 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg"] Nov 03 01:17:49.986315 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:49.986290 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa0de45_e9ac_42fd_9ab0_a353fa05b2e4.slice/crio-0ed6001fda0526431e4ed2cd22060881f061ba8905a21093435fa583734eae84 WatchSource:0}: Error finding container 0ed6001fda0526431e4ed2cd22060881f061ba8905a21093435fa583734eae84: Status 404 returned error can't find the container with id 0ed6001fda0526431e4ed2cd22060881f061ba8905a21093435fa583734eae84 Nov 03 01:17:50.000056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.000034 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ms9cn"] Nov 03 01:17:50.006775 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:50.006752 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502bb646_4048_4af0_90bb_a4f4688fe423.slice/crio-a17f1fbe38304d735ee0ad32fd62c8bb3c073a3b04240ff4ad571d290040c1f9 WatchSource:0}: Error finding container a17f1fbe38304d735ee0ad32fd62c8bb3c073a3b04240ff4ad571d290040c1f9: Status 404 returned error can't find the container with id a17f1fbe38304d735ee0ad32fd62c8bb3c073a3b04240ff4ad571d290040c1f9 Nov 03 01:17:50.022661 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:50.022631 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38df53d9_0193_43e6_a9a1_5ecd8fb2e456.slice/crio-aa4c0980424eeece05b6948cdfd6306a42d2f43f3d8b9f4022fd1c5fc33760c7 WatchSource:0}: Error finding container aa4c0980424eeece05b6948cdfd6306a42d2f43f3d8b9f4022fd1c5fc33760c7: Status 404 returned error can't find the container with id aa4c0980424eeece05b6948cdfd6306a42d2f43f3d8b9f4022fd1c5fc33760c7 Nov 03 01:17:50.023260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.023238 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ktk4w"] Nov 03 01:17:50.683976 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.683938 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ms9cn" event={"ID":"502bb646-4048-4af0-90bb-a4f4688fe423","Type":"ContainerStarted","Data":"a17f1fbe38304d735ee0ad32fd62c8bb3c073a3b04240ff4ad571d290040c1f9"} Nov 03 01:17:50.685443 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.685415 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktk4w" event={"ID":"38df53d9-0193-43e6-a9a1-5ecd8fb2e456","Type":"ContainerStarted","Data":"d97577853055231d6b8d04c29462280cc6ae8ef1ed916084d24e506b048dd096"} Nov 03 01:17:50.685552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.685465 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktk4w" event={"ID":"38df53d9-0193-43e6-a9a1-5ecd8fb2e456","Type":"ContainerStarted","Data":"aa4c0980424eeece05b6948cdfd6306a42d2f43f3d8b9f4022fd1c5fc33760c7"} Nov 03 01:17:50.686580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:50.686538 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" event={"ID":"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4","Type":"ContainerStarted","Data":"0ed6001fda0526431e4ed2cd22060881f061ba8905a21093435fa583734eae84"} Nov 03 01:17:51.946154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:51.946125 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:52.696307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:52.696256 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktk4w" event={"ID":"38df53d9-0193-43e6-a9a1-5ecd8fb2e456","Type":"ContainerStarted","Data":"3494bc16b05e62f88771c06353243cc07398a93355f1c21aa05901f427c92f2d"} Nov 03 01:17:53.476031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.475988 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476051 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476089 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476102 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476119 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476127 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c45667b98-7dq66: secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476147 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476175 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476200 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls podName:39aa5aec-9458-4a7e-928a-7d00a4f84182 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:01.476181376 +0000 UTC m=+17.055003964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls") pod "image-registry-c45667b98-7dq66" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182") : secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476220 4992 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476242 4992 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476254 4992 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769d849c4c-bmjrw: secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476256 4992 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476262 4992 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476282 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert podName:7ec03c17-824a-4f8e-9a62-c1df47789ec9 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:01.476265745 +0000 UTC m=+17.055088329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4frnh" (UID: "7ec03c17-824a-4f8e-9a62-c1df47789ec9") : secret "networking-console-plugin-cert" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476302 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls podName:b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:01.476291849 +0000 UTC m=+17.055114427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls") pod "image-registry-769d849c4c-bmjrw" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3") : secret "image-registry-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476316 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls podName:5bdb27b8-f440-43c6-a0fa-152432024154 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:01.476307951 +0000 UTC m=+17.055130534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-t82jm" (UID: "5bdb27b8-f440-43c6-a0fa-152432024154") : secret "samples-operator-tls" not found Nov 03 01:17:53.476500 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:17:53.476330 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs podName:fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc nodeName:}" failed. No retries permitted until 2026-11-03 01:18:01.476323159 +0000 UTC m=+17.055145737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs") pod "network-metrics-daemon-x9lj4" (UID: "fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc") : secret "metrics-daemon-secret" not found Nov 03 01:17:53.477167 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.476808 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835f0a42-f920-4d77-9178-4a174fe526c1-service-ca-bundle\") pod \"router-default-85ff878dc9-x2zvj\" (UID: \"835f0a42-f920-4d77-9178-4a174fe526c1\") " pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:53.784663 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:53.784514 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:54.082372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.082342 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:54.082579 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.082545 4992 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:54.301211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.301109 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85ff878dc9-x2zvj"] Nov 03 01:17:54.305000 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:17:54.304967 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835f0a42_f920_4d77_9178_4a174fe526c1.slice/crio-e89419a3ad0d74f266b8f06b9a52fdb9f4d5534dfe74abb41834e275785b5d74 WatchSource:0}: Error finding container e89419a3ad0d74f266b8f06b9a52fdb9f4d5534dfe74abb41834e275785b5d74: Status 404 returned error can't find the container with id e89419a3ad0d74f266b8f06b9a52fdb9f4d5534dfe74abb41834e275785b5d74 Nov 03 01:17:54.708009 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.707797 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ktk4w" event={"ID":"38df53d9-0193-43e6-a9a1-5ecd8fb2e456","Type":"ContainerStarted","Data":"81714c633b881d44c621b3cff6a529659488a558a3b7a8971b34b199dfa50e2d"} Nov 03 01:17:54.711184 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.711151 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" event={"ID":"835f0a42-f920-4d77-9178-4a174fe526c1","Type":"ContainerStarted","Data":"5f9a50e4128a475b1f687b07fb7451e49b6c732c22528cb7fddd1ed376259a98"} Nov 03 01:17:54.711319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.711203 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" event={"ID":"835f0a42-f920-4d77-9178-4a174fe526c1","Type":"ContainerStarted","Data":"e89419a3ad0d74f266b8f06b9a52fdb9f4d5534dfe74abb41834e275785b5d74"} Nov 03 01:17:54.713440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.713107 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-cs7fg" event={"ID":"3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4","Type":"ContainerStarted","Data":"9c6535f275d394ea3c86ce6fff37f58b03d11606e7ae3968ac5a2672cf7a5ee5"} Nov 03 01:17:54.715495 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.715468 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ms9cn" event={"ID":"502bb646-4048-4af0-90bb-a4f4688fe423","Type":"ContainerStarted","Data":"38cc06c29318f1e477cab11a3f7e23fa3367404cd6676dfb28a7cf7ef5cde471"} Nov 03 01:17:54.715615 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.715520 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ms9cn" event={"ID":"502bb646-4048-4af0-90bb-a4f4688fe423","Type":"ContainerStarted","Data":"af0ad9b553c1f4ed1e25712e503ee387a77729518af145f3ac1f36c60b8e6e3d"} Nov 03 01:17:54.715615 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.715577 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ms9cn" Nov 03 01:17:54.785190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.785158 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:54.787608 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:54.787586 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:55.215741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:55.215712 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8k5j6" Nov 03 01:17:56.722017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:56.721980 4992 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 01:17:57.687370 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:57.687336 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:57.688653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:57.688628 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85ff878dc9-x2zvj" Nov 03 01:17:58.588086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.588053 4992 scope.go:122] "RemoveContainer" containerID="fcb4a74c5e8b48f5cd962acb8ecf489b84d741721ea44b32b0675bd54029cf6b" Nov 03 01:17:58.732319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.732292 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:17:58.732476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.732413 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" event={"ID":"35bfbf47-81e9-4f0c-818c-965de586692b","Type":"ContainerStarted","Data":"235bb0eabebfad58b2e6710818d0cffefbed3654726c7faaeed49b1aa4cfea66"} Nov 03 01:17:58.732727 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.732700 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:17:58.733617 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.733592 4992 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-6jftg container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Nov 03 01:17:58.733711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:58.733648 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" podUID="35bfbf47-81e9-4f0c-818c-965de586692b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Nov 03 01:17:59.740502 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:17:59.740477 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6jftg" Nov 03 01:18:01.552230 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.552185 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:18:01.552747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.552241 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:18:01.552747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.552280 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:18:01.552747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.552310 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:01.552747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.552389 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:01.555215 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.555186 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc-metrics-certs\") pod \"network-metrics-daemon-x9lj4\" (UID: \"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc\") " pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:18:01.555405 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.555386 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5bdb27b8-f440-43c6-a0fa-152432024154-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-t82jm\" (UID: \"5bdb27b8-f440-43c6-a0fa-152432024154\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:18:01.555543 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.555520 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"image-registry-c45667b98-7dq66\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:01.555629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.555521 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"image-registry-769d849c4c-bmjrw\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:01.555629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.555600 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ec03c17-824a-4f8e-9a62-c1df47789ec9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4frnh\" (UID: \"7ec03c17-824a-4f8e-9a62-c1df47789ec9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:18:01.574383 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.574359 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" Nov 03 01:18:01.575476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.575450 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:01.576510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.576495 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" Nov 03 01:18:01.581540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.581527 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x9lj4" Nov 03 01:18:01.584693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.584669 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:01.967018 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:01.966989 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm"] Nov 03 01:18:02.015349 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:02.015321 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec03c17_824a_4f8e_9a62_c1df47789ec9.slice/crio-6b74207e0fc88d1e47fc6f88214cdd9d4cafd450d2bfbf06882de3c7ae64e758 WatchSource:0}: Error finding container 6b74207e0fc88d1e47fc6f88214cdd9d4cafd450d2bfbf06882de3c7ae64e758: Status 404 returned error can't find the container with id 6b74207e0fc88d1e47fc6f88214cdd9d4cafd450d2bfbf06882de3c7ae64e758 Nov 03 01:18:02.018970 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.018938 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4frnh"] Nov 03 01:18:02.019684 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.019663 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:18:02.050514 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:02.050491 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25b530f_3f98_4edd_97c4_e1ac1c3e5bb3.slice/crio-737247d2205e13076b93c6d4c8ed8e855b1d35afce5e3a10cea834897722e6f5 WatchSource:0}: Error finding container 737247d2205e13076b93c6d4c8ed8e855b1d35afce5e3a10cea834897722e6f5: Status 404 returned error can't find the container with id 737247d2205e13076b93c6d4c8ed8e855b1d35afce5e3a10cea834897722e6f5 Nov 03 01:18:02.051400 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:02.051344 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7b44a3_a1fd_4c6a_ae48_3bb70b9759fc.slice/crio-108d1c2c600d64b89601f783f5880ad3441499e2a5eff0ef6edaf40d35d858fd WatchSource:0}: Error finding container 108d1c2c600d64b89601f783f5880ad3441499e2a5eff0ef6edaf40d35d858fd: Status 404 returned error can't find the container with id 108d1c2c600d64b89601f783f5880ad3441499e2a5eff0ef6edaf40d35d858fd Nov 03 01:18:02.065680 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:02.065645 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aa5aec_9458_4a7e_928a_7d00a4f84182.slice/crio-a47605f003d19127978e98687a8314d71f199956fe768c4d87f6d862ec141591 WatchSource:0}: Error finding container a47605f003d19127978e98687a8314d71f199956fe768c4d87f6d862ec141591: Status 404 returned error can't find the container with id a47605f003d19127978e98687a8314d71f199956fe768c4d87f6d862ec141591 Nov 03 01:18:02.068465 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.068441 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x9lj4"] Nov 03 01:18:02.071046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.071025 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:18:02.150819 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:18:02.150777 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:18:02.748647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.748610 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9lj4" event={"ID":"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc","Type":"ContainerStarted","Data":"108d1c2c600d64b89601f783f5880ad3441499e2a5eff0ef6edaf40d35d858fd"} Nov 03 01:18:02.750104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.750080 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c45667b98-7dq66" event={"ID":"39aa5aec-9458-4a7e-928a-7d00a4f84182","Type":"ContainerStarted","Data":"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9"} Nov 03 01:18:02.750227 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.750121 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c45667b98-7dq66" event={"ID":"39aa5aec-9458-4a7e-928a-7d00a4f84182","Type":"ContainerStarted","Data":"a47605f003d19127978e98687a8314d71f199956fe768c4d87f6d862ec141591"} Nov 03 01:18:02.750282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.750231 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:02.751532 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.751511 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" event={"ID":"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3","Type":"ContainerStarted","Data":"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb"} Nov 03 01:18:02.751658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.751553 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:02.751658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.751594 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" event={"ID":"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3","Type":"ContainerStarted","Data":"737247d2205e13076b93c6d4c8ed8e855b1d35afce5e3a10cea834897722e6f5"} Nov 03 01:18:02.752653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.752628 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" event={"ID":"5bdb27b8-f440-43c6-a0fa-152432024154","Type":"ContainerStarted","Data":"d111e0e489f0b93bbde34423570f812e0d46ad7972fb0a7b6a87da29cbbd1a93"} Nov 03 01:18:02.753622 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:02.753604 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" event={"ID":"7ec03c17-824a-4f8e-9a62-c1df47789ec9","Type":"ContainerStarted","Data":"6b74207e0fc88d1e47fc6f88214cdd9d4cafd450d2bfbf06882de3c7ae64e758"} Nov 03 01:18:04.093711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.093676 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh"] Nov 03 01:18:04.097687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.097655 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:04.136643 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.136607 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:18:04.160190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.160154 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hwck6\"" Nov 03 01:18:04.160190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.160155 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 01:18:04.177639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.177604 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/faa8c85c-155f-4121-970b-9ea01d3188b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zb5gh\" (UID: \"faa8c85c-155f-4121-970b-9ea01d3188b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:04.189022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.188989 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-4xbts"] Nov 03 01:18:04.192695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.192673 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.207328 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.207298 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644d8d86f7-q58gd"] Nov 03 01:18:04.210930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.210906 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.212574 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.212540 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh"] Nov 03 01:18:04.249714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.249625 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 01:18:04.249714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.249655 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 01:18:04.257239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.257206 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jfhh5\"" Nov 03 01:18:04.260856 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.260830 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4xbts"] Nov 03 01:18:04.263885 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.263827 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644d8d86f7-q58gd"] Nov 03 01:18:04.278804 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.278776 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fr42\" (UniqueName: \"kubernetes.io/projected/5466080e-958b-4f26-9ead-e493fd972a1a-kube-api-access-2fr42\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.278926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.278820 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/faa8c85c-155f-4121-970b-9ea01d3188b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zb5gh\" (UID: \"faa8c85c-155f-4121-970b-9ea01d3188b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:04.278926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.278847 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qgj\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-kube-api-access-64qgj\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279122 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279095 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5466080e-958b-4f26-9ead-e493fd972a1a-tmp\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.279233 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279146 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-tls\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279233 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279178 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-certificates\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279348 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279276 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-installation-pull-secrets\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279348 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279321 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-image-registry-private-configuration\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279348 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-trusted-ca\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279410 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/976a95b1-48ca-4bd1-b807-7970adb1c53b-ca-trust-extracted\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.279438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.279432 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-bound-sa-token\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.281896 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.281864 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/faa8c85c-155f-4121-970b-9ea01d3188b9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-zb5gh\" (UID: \"faa8c85c-155f-4121-970b-9ea01d3188b9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:04.380605 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380571 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fr42\" (UniqueName: \"kubernetes.io/projected/5466080e-958b-4f26-9ead-e493fd972a1a-kube-api-access-2fr42\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.380768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380620 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qgj\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-kube-api-access-64qgj\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.380768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380637 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5466080e-958b-4f26-9ead-e493fd972a1a-tmp\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.380768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380657 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-tls\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.380768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380715 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-certificates\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.380768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380767 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-installation-pull-secrets\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380796 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-image-registry-private-configuration\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380926 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-trusted-ca\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.380979 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/976a95b1-48ca-4bd1-b807-7970adb1c53b-ca-trust-extracted\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.381006 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-bound-sa-token\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381278 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.381101 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5466080e-958b-4f26-9ead-e493fd972a1a-tmp\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.381428 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.381410 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/976a95b1-48ca-4bd1-b807-7970adb1c53b-ca-trust-extracted\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.381712 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.381684 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-certificates\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.382130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.382106 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/976a95b1-48ca-4bd1-b807-7970adb1c53b-trusted-ca\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.383382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.383356 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-image-registry-private-configuration\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.383670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.383643 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/976a95b1-48ca-4bd1-b807-7970adb1c53b-installation-pull-secrets\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.383767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.383752 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-registry-tls\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.414152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.414125 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:04.443427 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.443398 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fr42\" (UniqueName: \"kubernetes.io/projected/5466080e-958b-4f26-9ead-e493fd972a1a-kube-api-access-2fr42\") pod \"downloads-6dd789c8b7-4xbts\" (UID: \"5466080e-958b-4f26-9ead-e493fd972a1a\") " pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.443690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.443671 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qgj\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-kube-api-access-64qgj\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.443829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.443802 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/976a95b1-48ca-4bd1-b807-7970adb1c53b-bound-sa-token\") pod \"image-registry-644d8d86f7-q58gd\" (UID: \"976a95b1-48ca-4bd1-b807-7970adb1c53b\") " pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.506916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.506888 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:04.523901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.523647 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:04.721207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.721163 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ms9cn" Nov 03 01:18:04.765432 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.765349 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" event={"ID":"5bdb27b8-f440-43c6-a0fa-152432024154","Type":"ContainerStarted","Data":"02c0cd527787c3cad53fc245131e1b42453ee19c0a16933da3d332e74790ce7c"} Nov 03 01:18:04.765732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.765701 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-t82jm" event={"ID":"5bdb27b8-f440-43c6-a0fa-152432024154","Type":"ContainerStarted","Data":"e97f56109a79dcbdf103bd29e82f9436b07d195f24077d00678bbdcfbdc4fdf3"} Nov 03 01:18:04.767443 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.767409 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4frnh" event={"ID":"7ec03c17-824a-4f8e-9a62-c1df47789ec9","Type":"ContainerStarted","Data":"e3f0d217765499651ebbd9d8d4650882e1f2e51a85a8fc27e666e11647d61efb"} Nov 03 01:18:04.790336 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:04.790305 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa8c85c_155f_4121_970b_9ea01d3188b9.slice/crio-b16e01e18393a62c2e0b675d3f52a48467ffab630a780df2e954f5c1b556a2fd WatchSource:0}: Error finding container b16e01e18393a62c2e0b675d3f52a48467ffab630a780df2e954f5c1b556a2fd: Status 404 returned error can't find the container with id b16e01e18393a62c2e0b675d3f52a48467ffab630a780df2e954f5c1b556a2fd Nov 03 01:18:04.791085 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.791061 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh"] Nov 03 01:18:04.890291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.890264 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-4xbts"] Nov 03 01:18:04.901899 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:04.901858 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5466080e_958b_4f26_9ead_e493fd972a1a.slice/crio-db84279de1ebd0f2862139787a60ed91bf73ce7e55f577f02b3578af7a21f544 WatchSource:0}: Error finding container db84279de1ebd0f2862139787a60ed91bf73ce7e55f577f02b3578af7a21f544: Status 404 returned error can't find the container with id db84279de1ebd0f2862139787a60ed91bf73ce7e55f577f02b3578af7a21f544 Nov 03 01:18:04.942336 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:04.942304 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976a95b1_48ca_4bd1_b807_7970adb1c53b.slice/crio-b0aff0e21bbbd64bc7e1410cdbe3e42c737c50409ca02184a9cdb150b5dc0615 WatchSource:0}: Error finding container b0aff0e21bbbd64bc7e1410cdbe3e42c737c50409ca02184a9cdb150b5dc0615: Status 404 returned error can't find the container with id b0aff0e21bbbd64bc7e1410cdbe3e42c737c50409ca02184a9cdb150b5dc0615 Nov 03 01:18:04.942568 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:04.942540 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644d8d86f7-q58gd"] Nov 03 01:18:05.773475 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.773432 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" event={"ID":"faa8c85c-155f-4121-970b-9ea01d3188b9","Type":"ContainerStarted","Data":"b16e01e18393a62c2e0b675d3f52a48467ffab630a780df2e954f5c1b556a2fd"} Nov 03 01:18:05.775240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.775210 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" event={"ID":"976a95b1-48ca-4bd1-b807-7970adb1c53b","Type":"ContainerStarted","Data":"e1968ae7b24d197b81f57841a827b0e7a6d932c177da3c96d38d3584b002b1b5"} Nov 03 01:18:05.775364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.775253 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" event={"ID":"976a95b1-48ca-4bd1-b807-7970adb1c53b","Type":"ContainerStarted","Data":"b0aff0e21bbbd64bc7e1410cdbe3e42c737c50409ca02184a9cdb150b5dc0615"} Nov 03 01:18:05.775364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.775294 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:05.776453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.776430 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4xbts" event={"ID":"5466080e-958b-4f26-9ead-e493fd972a1a","Type":"ContainerStarted","Data":"db84279de1ebd0f2862139787a60ed91bf73ce7e55f577f02b3578af7a21f544"} Nov 03 01:18:05.778150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.778126 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9lj4" event={"ID":"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc","Type":"ContainerStarted","Data":"d8f62dcafae31dc0e833c74e630c286879d0af01328ad09bbb4be1fe8182b2a2"} Nov 03 01:18:05.778259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.778170 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x9lj4" event={"ID":"fb7b44a3-a1fd-4c6a-ae48-3bb70b9759fc","Type":"ContainerStarted","Data":"b112d119b7556da4bcaa19387ec14eae2e48a515b4ac2e2f96dd0295519b04cd"} Nov 03 01:18:05.900600 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:05.900527 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" podStartSLOduration=1.9005135119999998 podStartE2EDuration="1.900513512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:05.899621736 +0000 UTC m=+21.478444322" watchObservedRunningTime="2026-11-03 01:18:05.900513512 +0000 UTC m=+21.479336109" Nov 03 01:18:06.784997 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:06.784958 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" event={"ID":"faa8c85c-155f-4121-970b-9ea01d3188b9","Type":"ContainerStarted","Data":"157fa3157dc84244b3de01cdf1b8549ae35282ae8c75b8960acc56d3ff09eb15"} Nov 03 01:18:06.785463 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:06.785405 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:06.790777 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:06.790754 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" Nov 03 01:18:06.904872 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:06.904704 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-zb5gh" podStartSLOduration=1.140856225 podStartE2EDuration="2.90468511s" totalImagesPullingTime="1.763828885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:06.904011511 +0000 UTC m=+22.482834105" watchObservedRunningTime="2026-11-03 01:18:06.90468511 +0000 UTC m=+22.483507698" Nov 03 01:18:07.845513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.845484 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x548w"] Nov 03 01:18:07.852948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.852922 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:07.904810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.904773 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x548w"] Nov 03 01:18:07.911649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.911622 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:07.911795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.911676 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:07.911795 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.911773 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:07.911906 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.911884 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65r2\" (UniqueName: \"kubernetes.io/projected/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-kube-api-access-x65r2\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:07.959335 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.959300 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g2k22\"" Nov 03 01:18:07.959508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.959328 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 01:18:07.959508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.959337 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 01:18:07.959508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:07.959398 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 01:18:08.013130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.013089 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.013323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.013153 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65r2\" (UniqueName: \"kubernetes.io/projected/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-kube-api-access-x65r2\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.013323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.013209 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.013323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.013262 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.013918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.013870 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.016019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.015994 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.016119 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.016068 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.076145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.076113 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65r2\" (UniqueName: \"kubernetes.io/projected/f1fc01e7-33b8-4129-a0f5-16ebe5c9786d-kube-api-access-x65r2\") pod \"prometheus-operator-9cd89b74c-x548w\" (UID: \"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.167227 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.167192 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" Nov 03 01:18:08.540368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.540290 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-x548w"] Nov 03 01:18:08.569371 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:08.569338 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc01e7_33b8_4129_a0f5_16ebe5c9786d.slice/crio-0182016d3c4fd1d97d7d80abef38b5db884af288e1c95d8cc73aeab4ef891f64 WatchSource:0}: Error finding container 0182016d3c4fd1d97d7d80abef38b5db884af288e1c95d8cc73aeab4ef891f64: Status 404 returned error can't find the container with id 0182016d3c4fd1d97d7d80abef38b5db884af288e1c95d8cc73aeab4ef891f64 Nov 03 01:18:08.795283 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:08.795203 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" event={"ID":"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d","Type":"ContainerStarted","Data":"0182016d3c4fd1d97d7d80abef38b5db884af288e1c95d8cc73aeab4ef891f64"} Nov 03 01:18:11.811706 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:11.811668 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" event={"ID":"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d","Type":"ContainerStarted","Data":"319d2ae4052db2d9f39b651063af3999af2e709be8b676331290682cc799bdf9"} Nov 03 01:18:11.812192 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:11.811728 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" event={"ID":"f1fc01e7-33b8-4129-a0f5-16ebe5c9786d","Type":"ContainerStarted","Data":"aa134f0120a98437a381980a4ce9afe00a618e5d664ea5ec8e08d9ce910e88fd"} Nov 03 01:18:11.932360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:11.932303 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-x548w" podStartSLOduration=2.6596859139999998 podStartE2EDuration="4.932282982s" totalImagesPullingTime="2.272597068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:11.931803863 +0000 UTC m=+27.510626451" watchObservedRunningTime="2026-11-03 01:18:11.932282982 +0000 UTC m=+27.511105570" Nov 03 01:18:15.505279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.505241 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nzg77"] Nov 03 01:18:15.543499 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.543465 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.583636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.583600 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-root\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.583796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.583663 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-textfile\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.583854 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.583831 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.583896 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.583882 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-tls\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.583976 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.583960 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-metrics-client-ca\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.584018 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.584003 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-wtmp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.584058 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.584021 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.584110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.584092 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-sys\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.584143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.584123 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bjp\" (UniqueName: \"kubernetes.io/projected/3930af2f-52ea-435f-8ae9-34678e6040c9-kube-api-access-n7bjp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685280 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-metrics-client-ca\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685335 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-wtmp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685440 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685490 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-wtmp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685492 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-sys\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685534 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-sys\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685592 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bjp\" (UniqueName: \"kubernetes.io/projected/3930af2f-52ea-435f-8ae9-34678e6040c9-kube-api-access-n7bjp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685627 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-root\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685677 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-textfile\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685702 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3930af2f-52ea-435f-8ae9-34678e6040c9-root\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685723 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.685955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.685770 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-tls\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.686110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.686000 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-textfile\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.686468 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.686444 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-metrics-client-ca\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.714931 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.714904 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 01:18:15.715110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.715097 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w7g4c\"" Nov 03 01:18:15.715224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.715114 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 01:18:15.715224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.715128 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 01:18:15.716548 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.716528 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-accelerators-collector-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.718443 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.718425 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-tls\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.718443 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.718439 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3930af2f-52ea-435f-8ae9-34678e6040c9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.747979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.747942 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bjp\" (UniqueName: \"kubernetes.io/projected/3930af2f-52ea-435f-8ae9-34678e6040c9-kube-api-access-n7bjp\") pod \"node-exporter-nzg77\" (UID: \"3930af2f-52ea-435f-8ae9-34678e6040c9\") " pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.909286 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:15.909208 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nzg77" Nov 03 01:18:15.942465 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:18:15.942427 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3930af2f_52ea_435f_8ae9_34678e6040c9.slice/crio-0e3f232d8135fb3f61d677b644c752c1fd7251b0323656deb41aa6473e016538 WatchSource:0}: Error finding container 0e3f232d8135fb3f61d677b644c752c1fd7251b0323656deb41aa6473e016538: Status 404 returned error can't find the container with id 0e3f232d8135fb3f61d677b644c752c1fd7251b0323656deb41aa6473e016538 Nov 03 01:18:16.835214 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:16.835173 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzg77" event={"ID":"3930af2f-52ea-435f-8ae9-34678e6040c9","Type":"ContainerStarted","Data":"0e3f232d8135fb3f61d677b644c752c1fd7251b0323656deb41aa6473e016538"} Nov 03 01:18:17.352648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:17.352616 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qk55d" Nov 03 01:18:19.851430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:19.851348 4992 generic.go:356] "Generic (PLEG): container finished" podID="3930af2f-52ea-435f-8ae9-34678e6040c9" containerID="b227959bc4aad5522ce300e7ef33c92d8b5f7bfa1d28c92ea7e209942523c496" exitCode=0 Nov 03 01:18:19.851858 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:19.851427 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzg77" event={"ID":"3930af2f-52ea-435f-8ae9-34678e6040c9","Type":"ContainerDied","Data":"b227959bc4aad5522ce300e7ef33c92d8b5f7bfa1d28c92ea7e209942523c496"} Nov 03 01:18:20.857736 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:20.857703 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzg77" event={"ID":"3930af2f-52ea-435f-8ae9-34678e6040c9","Type":"ContainerStarted","Data":"e59974bf18d196121a2c780239ccb26ee05b2e2d3ee0aa3dccea77fca48c421a"} Nov 03 01:18:20.858122 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:20.857760 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nzg77" event={"ID":"3930af2f-52ea-435f-8ae9-34678e6040c9","Type":"ContainerStarted","Data":"8724953eb042251ec66b33336dacc8f1cbf7630611cbfeac2a3c6d4fa7290c51"} Nov 03 01:18:21.032603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:21.032529 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nzg77" podStartSLOduration=2.343451175 podStartE2EDuration="6.032515187s" totalImagesPullingTime="3.601469545s" totalInitContainerRuntime="87.594467ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:21.032272328 +0000 UTC m=+36.611094914" watchObservedRunningTime="2026-11-03 01:18:21.032515187 +0000 UTC m=+36.611337773" Nov 03 01:18:23.764197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:23.764166 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:24.773808 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:24.773728 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:26.789642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:26.789609 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-644d8d86f7-q58gd" Nov 03 01:18:27.019102 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:27.019071 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:18:29.788670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:29.788624 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" podUID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" containerName="registry" containerID="cri-o://4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb" gracePeriod=30 Nov 03 01:18:30.069086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.069060 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:30.218194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218161 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218215 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218267 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218306 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218339 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218366 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218410 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218464 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") pod \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\" (UID: \"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3\") " Nov 03 01:18:30.218783 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218744 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:30.218856 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.218825 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:30.221215 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.221183 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:30.221368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.221349 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:30.223825 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.223797 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "kube-api-access-6nb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:30.226173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.226132 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:30.227220 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.227127 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:30.227436 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.227416 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration" pod "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" (UID: "b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320003 4992 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-ca-trust-extracted\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320030 4992 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-image-registry-private-configuration\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320041 4992 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-bound-sa-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320051 4992 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320062 4992 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-installation-pull-secrets\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320072 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nb5g\" (UniqueName: \"kubernetes.io/projected/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-kube-api-access-6nb5g\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320082 4992 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-registry-certificates\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.320216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.320090 4992 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3-trusted-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:30.903756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.903672 4992 generic.go:356] "Generic (PLEG): container finished" podID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" containerID="4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb" exitCode=0 Nov 03 01:18:30.903756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.903737 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" Nov 03 01:18:30.904206 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.903771 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" event={"ID":"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3","Type":"ContainerDied","Data":"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb"} Nov 03 01:18:30.904206 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.903864 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769d849c4c-bmjrw" event={"ID":"b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3","Type":"ContainerDied","Data":"737247d2205e13076b93c6d4c8ed8e855b1d35afce5e3a10cea834897722e6f5"} Nov 03 01:18:30.904206 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.903885 4992 scope.go:122] "RemoveContainer" containerID="4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb" Nov 03 01:18:30.913290 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.913270 4992 scope.go:122] "RemoveContainer" containerID="4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb" Nov 03 01:18:30.913550 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:18:30.913521 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb\": container with ID starting with 4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb not found: ID does not exist" containerID="4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb" Nov 03 01:18:30.913634 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:30.913572 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb"} err="failed to get container status \"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb\": rpc error: code = NotFound desc = could not find container \"4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb\": container with ID starting with 4ba0f5dc0dca557a8c8d3e7747eaedc6680430dec24ed9d8ff30d84edad4c8bb not found: ID does not exist" Nov 03 01:18:31.127615 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:31.127577 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:18:31.179340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:31.179302 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-769d849c4c-bmjrw"] Nov 03 01:18:32.152517 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:18:32.152480 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:18:32.593921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:32.593885 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" path="/var/lib/kubelet/pods/b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3/volumes" Nov 03 01:18:45.970275 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:45.970234 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-4xbts" event={"ID":"5466080e-958b-4f26-9ead-e493fd972a1a","Type":"ContainerStarted","Data":"8dabd10d98c2f3a8928a80142293ac6acfb0ca759aa61cd2af200306c9bfd6ad"} Nov 03 01:18:45.970853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:45.970479 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:45.981144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:45.981113 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-4xbts" Nov 03 01:18:46.142401 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:46.142344 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-4xbts" podStartSLOduration=1.618681375 podStartE2EDuration="42.142331171s" totalImagesPullingTime="40.523649796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:46.142262468 +0000 UTC m=+61.721085056" watchObservedRunningTime="2026-11-03 01:18:46.142331171 +0000 UTC m=+61.721153754" Nov 03 01:18:52.039732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.039685 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c45667b98-7dq66" podUID="39aa5aec-9458-4a7e-928a-7d00a4f84182" containerName="registry" containerID="cri-o://711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9" gracePeriod=30 Nov 03 01:18:52.349658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.349627 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:52.409622 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409578 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.409833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409643 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.409833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409682 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.409833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409722 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.409833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409792 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.409833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409823 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.410098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409868 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.410098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.409903 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") pod \"39aa5aec-9458-4a7e-928a-7d00a4f84182\" (UID: \"39aa5aec-9458-4a7e-928a-7d00a4f84182\") " Nov 03 01:18:52.410963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.410921 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:52.413388 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.413359 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.413510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.413396 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.413740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.413705 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "kube-api-access-p96xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.414164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.414135 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.414819 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.414793 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:52.420648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.420497 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.423970 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.423947 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls" pod "39aa5aec-9458-4a7e-928a-7d00a4f84182" (UID: "39aa5aec-9458-4a7e-928a-7d00a4f84182"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.510487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510450 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p96xj\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-kube-api-access-p96xj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510482 4992 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-image-registry-private-configuration\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510493 4992 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-certificates\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510504 4992 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-registry-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510513 4992 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39aa5aec-9458-4a7e-928a-7d00a4f84182-ca-trust-extracted\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510526 4992 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39aa5aec-9458-4a7e-928a-7d00a4f84182-bound-sa-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510537 4992 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39aa5aec-9458-4a7e-928a-7d00a4f84182-trusted-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.510821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:52.510546 4992 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39aa5aec-9458-4a7e-928a-7d00a4f84182-installation-pull-secrets\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:18:53.006687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.006647 4992 generic.go:356] "Generic (PLEG): container finished" podID="39aa5aec-9458-4a7e-928a-7d00a4f84182" containerID="711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9" exitCode=0 Nov 03 01:18:53.006905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.006699 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c45667b98-7dq66" Nov 03 01:18:53.006905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.006721 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c45667b98-7dq66" event={"ID":"39aa5aec-9458-4a7e-928a-7d00a4f84182","Type":"ContainerDied","Data":"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9"} Nov 03 01:18:53.006905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.006771 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c45667b98-7dq66" event={"ID":"39aa5aec-9458-4a7e-928a-7d00a4f84182","Type":"ContainerDied","Data":"a47605f003d19127978e98687a8314d71f199956fe768c4d87f6d862ec141591"} Nov 03 01:18:53.006905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.006800 4992 scope.go:122] "RemoveContainer" containerID="711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9" Nov 03 01:18:53.016585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.016541 4992 scope.go:122] "RemoveContainer" containerID="711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9" Nov 03 01:18:53.017033 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:18:53.017007 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9\": container with ID starting with 711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9 not found: ID does not exist" containerID="711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9" Nov 03 01:18:53.017130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.017043 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9"} err="failed to get container status \"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9\": rpc error: code = NotFound desc = could not find container \"711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9\": container with ID starting with 711670f9be0131a28c7d6c7cc8ccf9913cc28d2e250d28fd3bf9e0425741cfe9 not found: ID does not exist" Nov 03 01:18:53.176671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.176640 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:18:53.228471 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:53.228437 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c45667b98-7dq66"] Nov 03 01:18:54.593554 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:54.593522 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39aa5aec-9458-4a7e-928a-7d00a4f84182" path="/var/lib/kubelet/pods/39aa5aec-9458-4a7e-928a-7d00a4f84182/volumes" Nov 03 01:18:56.026531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:56.026496 4992 generic.go:356] "Generic (PLEG): container finished" podID="477e52dd-4c41-4415-ada1-414b13751aef" containerID="b5a22c92185ca93cc6d2aff327d8659b3519bbbe84c129469ec81ec6fb223095" exitCode=0 Nov 03 01:18:56.027104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:56.026598 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" event={"ID":"477e52dd-4c41-4415-ada1-414b13751aef","Type":"ContainerDied","Data":"b5a22c92185ca93cc6d2aff327d8659b3519bbbe84c129469ec81ec6fb223095"} Nov 03 01:18:56.027104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:56.026929 4992 scope.go:122] "RemoveContainer" containerID="b5a22c92185ca93cc6d2aff327d8659b3519bbbe84c129469ec81ec6fb223095" Nov 03 01:18:57.035288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:18:57.035245 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6x5ln" event={"ID":"477e52dd-4c41-4415-ada1-414b13751aef","Type":"ContainerStarted","Data":"c9d0f3b936bc5af4b54c844afb54ec150ba95648ccce2a9c9573b728fe5d910c"} Nov 03 01:19:02.149240 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:19:02.149201 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:19:08.078238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:08.078201 4992 generic.go:356] "Generic (PLEG): container finished" podID="efde973b-4882-427d-a08a-b0b14ef8aaa0" containerID="bfee83737b05f00e80b08b4e1558da12333f438e68a5d8ff672e2498ab819e94" exitCode=0 Nov 03 01:19:08.078665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:08.078274 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" event={"ID":"efde973b-4882-427d-a08a-b0b14ef8aaa0","Type":"ContainerDied","Data":"bfee83737b05f00e80b08b4e1558da12333f438e68a5d8ff672e2498ab819e94"} Nov 03 01:19:09.083113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.083082 4992 generic.go:356] "Generic (PLEG): container finished" podID="8895329f-6ed6-41c8-9f6b-d268dff1c4dc" containerID="3fe95386013e3b0c7dd144e2ecc6b771ba6f8a8667efd63780dd68e5da6e7927" exitCode=0 Nov 03 01:19:09.083509 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.083159 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" event={"ID":"8895329f-6ed6-41c8-9f6b-d268dff1c4dc","Type":"ContainerDied","Data":"3fe95386013e3b0c7dd144e2ecc6b771ba6f8a8667efd63780dd68e5da6e7927"} Nov 03 01:19:09.083844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.083572 4992 scope.go:122] "RemoveContainer" containerID="3fe95386013e3b0c7dd144e2ecc6b771ba6f8a8667efd63780dd68e5da6e7927" Nov 03 01:19:09.257168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.257133 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:19:09.336568 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336478 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") pod \"efde973b-4882-427d-a08a-b0b14ef8aaa0\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " Nov 03 01:19:09.336568 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336518 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") pod \"efde973b-4882-427d-a08a-b0b14ef8aaa0\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " Nov 03 01:19:09.336775 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336580 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") pod \"efde973b-4882-427d-a08a-b0b14ef8aaa0\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " Nov 03 01:19:09.336775 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336656 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") pod \"efde973b-4882-427d-a08a-b0b14ef8aaa0\" (UID: \"efde973b-4882-427d-a08a-b0b14ef8aaa0\") " Nov 03 01:19:09.336954 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336925 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle" pod "efde973b-4882-427d-a08a-b0b14ef8aaa0" (UID: "efde973b-4882-427d-a08a-b0b14ef8aaa0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:09.337008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.336946 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path" pod "efde973b-4882-427d-a08a-b0b14ef8aaa0" (UID: "efde973b-4882-427d-a08a-b0b14ef8aaa0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:09.337073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.337007 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle" pod "efde973b-4882-427d-a08a-b0b14ef8aaa0" (UID: "efde973b-4882-427d-a08a-b0b14ef8aaa0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:09.338996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.338974 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn" pod "efde973b-4882-427d-a08a-b0b14ef8aaa0" (UID: "efde973b-4882-427d-a08a-b0b14ef8aaa0"). InnerVolumeSpecName "kube-api-access-wl8nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:09.437497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.437458 4992 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/efde973b-4882-427d-a08a-b0b14ef8aaa0-archives-path\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.437497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.437493 4992 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-service-ca-bundle\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.437730 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.437507 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8nn\" (UniqueName: \"kubernetes.io/projected/efde973b-4882-427d-a08a-b0b14ef8aaa0-kube-api-access-wl8nn\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.437730 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:09.437521 4992 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efde973b-4882-427d-a08a-b0b14ef8aaa0-trusted-ca-bundle\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:19:10.089176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:10.089139 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-rbtvp" event={"ID":"8895329f-6ed6-41c8-9f6b-d268dff1c4dc","Type":"ContainerStarted","Data":"711bb78d19d1c5527b1cb0e7a351061433aba3e82175ab3ed9ca15fcd7c3fa14"} Nov 03 01:19:10.090409 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:10.090390 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" Nov 03 01:19:10.090409 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:10.090403 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-sdbcx-x69tk" event={"ID":"efde973b-4882-427d-a08a-b0b14ef8aaa0","Type":"ContainerDied","Data":"1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1"} Nov 03 01:19:10.090613 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:10.090434 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f268fa63406db8b72b2d8f934882dfcd24f647306a6a7390b8bb08c99986eb1" Nov 03 01:19:17.119573 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:17.119527 4992 generic.go:356] "Generic (PLEG): container finished" podID="cacd2582-2d59-4dc3-ba1f-28118cb27609" containerID="8492efaa6bd7ae9538de7e3e37ef7c24e8138bb2c4c173dd06c0c945ba45e8e7" exitCode=0 Nov 03 01:19:17.119986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:17.119596 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" event={"ID":"cacd2582-2d59-4dc3-ba1f-28118cb27609","Type":"ContainerDied","Data":"8492efaa6bd7ae9538de7e3e37ef7c24e8138bb2c4c173dd06c0c945ba45e8e7"} Nov 03 01:19:17.119986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:17.119914 4992 scope.go:122] "RemoveContainer" containerID="8492efaa6bd7ae9538de7e3e37ef7c24e8138bb2c4c173dd06c0c945ba45e8e7" Nov 03 01:19:18.125375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.125341 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tvs7q" event={"ID":"cacd2582-2d59-4dc3-ba1f-28118cb27609","Type":"ContainerStarted","Data":"6eee4b0ecf650d29120cf300179fa5d45dc9dee397e36e7efd9c7a09d4af3aba"} Nov 03 01:19:18.459302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459271 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ptwht-p8b49"] Nov 03 01:19:18.459625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459612 4992 cpu_manager.go:414] "containerMap: removing container" podUID="efde973b-4882-427d-a08a-b0b14ef8aaa0" containerName="insights-gathering" Nov 03 01:19:18.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459625 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde973b-4882-427d-a08a-b0b14ef8aaa0" containerName="insights-gathering" Nov 03 01:19:18.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459637 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" containerName="registry" Nov 03 01:19:18.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459642 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" containerName="registry" Nov 03 01:19:18.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459664 4992 cpu_manager.go:414] "containerMap: removing container" podUID="39aa5aec-9458-4a7e-928a-7d00a4f84182" containerName="registry" Nov 03 01:19:18.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459670 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39aa5aec-9458-4a7e-928a-7d00a4f84182" containerName="registry" Nov 03 01:19:18.459823 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459722 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde973b-4882-427d-a08a-b0b14ef8aaa0" containerName="insights-gathering" Nov 03 01:19:18.459823 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459731 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25b530f-3f98-4edd-97c4-e1ac1c3e5bb3" containerName="registry" Nov 03 01:19:18.459823 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.459741 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="39aa5aec-9458-4a7e-928a-7d00a4f84182" containerName="registry" Nov 03 01:19:18.472086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.472057 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.513923 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.513837 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.513923 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.513889 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hq7n\" (UniqueName: \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.514197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.513992 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.514197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.514050 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.516640 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.516611 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ptwht-p8b49"] Nov 03 01:19:18.526595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.526552 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-27r6g\"" Nov 03 01:19:18.615201 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.615169 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.615384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.615229 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hq7n\" (UniqueName: \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.615665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.615628 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.615850 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.615823 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.615977 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.615955 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.616225 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.616202 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.616311 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.616294 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.678670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.678632 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hq7n\" (UniqueName: \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\") pod \"periodic-gathering-ptwht-p8b49\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:18.803818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:18.803740 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:19:19.186645 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:19:19.186608 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c069af_b064_47d5_ac44_b8ab45175489.slice/crio-a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74 WatchSource:0}: Error finding container a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74: Status 404 returned error can't find the container with id a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74 Nov 03 01:19:19.188047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:19.188018 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ptwht-p8b49"] Nov 03 01:19:20.134651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:20.134614 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ptwht-p8b49" event={"ID":"f0c069af-b064-47d5-ac44-b8ab45175489","Type":"ContainerStarted","Data":"03cafa7979853b7e8bae444a6da530ee05621c74b1aa2ddd2cfe24034ffa3a93"} Nov 03 01:19:20.134861 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:20.134666 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ptwht-p8b49" event={"ID":"f0c069af-b064-47d5-ac44-b8ab45175489","Type":"ContainerStarted","Data":"a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74"} Nov 03 01:19:20.266051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:19:20.265995 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ptwht-p8b49" podStartSLOduration=2.265980789 podStartE2EDuration="2.265980789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:20.264586585 +0000 UTC m=+95.843409165" watchObservedRunningTime="2026-11-03 01:19:20.265980789 +0000 UTC m=+95.844803376" Nov 03 01:19:32.149220 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:19:32.149177 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:20:02.149066 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:20:02.148977 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:20:32.149284 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:20:32.149241 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:20:44.483608 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:44.483574 4992 generic.go:356] "Generic (PLEG): container finished" podID="f0c069af-b064-47d5-ac44-b8ab45175489" containerID="03cafa7979853b7e8bae444a6da530ee05621c74b1aa2ddd2cfe24034ffa3a93" exitCode=0 Nov 03 01:20:44.483994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:44.483631 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ptwht-p8b49" event={"ID":"f0c069af-b064-47d5-ac44-b8ab45175489","Type":"ContainerDied","Data":"03cafa7979853b7e8bae444a6da530ee05621c74b1aa2ddd2cfe24034ffa3a93"} Nov 03 01:20:45.608149 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.608125 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:20:45.705791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.705753 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\") pod \"f0c069af-b064-47d5-ac44-b8ab45175489\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " Nov 03 01:20:45.705961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.705818 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\") pod \"f0c069af-b064-47d5-ac44-b8ab45175489\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " Nov 03 01:20:45.705961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.705878 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\") pod \"f0c069af-b064-47d5-ac44-b8ab45175489\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " Nov 03 01:20:45.705961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.705921 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\" (UniqueName: \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\") pod \"f0c069af-b064-47d5-ac44-b8ab45175489\" (UID: \"f0c069af-b064-47d5-ac44-b8ab45175489\") " Nov 03 01:20:45.706190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.706158 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path" pod "f0c069af-b064-47d5-ac44-b8ab45175489" (UID: "f0c069af-b064-47d5-ac44-b8ab45175489"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:45.706235 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.706209 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle" pod "f0c069af-b064-47d5-ac44-b8ab45175489" (UID: "f0c069af-b064-47d5-ac44-b8ab45175489"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:45.706272 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.706224 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle" pod "f0c069af-b064-47d5-ac44-b8ab45175489" (UID: "f0c069af-b064-47d5-ac44-b8ab45175489"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:45.708092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.708069 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n" pod "f0c069af-b064-47d5-ac44-b8ab45175489" (UID: "f0c069af-b064-47d5-ac44-b8ab45175489"). InnerVolumeSpecName "kube-api-access-7hq7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:45.806821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.806724 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hq7n\" (UniqueName: \"kubernetes.io/projected/f0c069af-b064-47d5-ac44-b8ab45175489-kube-api-access-7hq7n\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:45.806821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.806760 4992 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-trusted-ca-bundle\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:45.806821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.806774 4992 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0c069af-b064-47d5-ac44-b8ab45175489-archives-path\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:45.806821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:45.806788 4992 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0c069af-b064-47d5-ac44-b8ab45175489-service-ca-bundle\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:20:46.492137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:46.492103 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ptwht-p8b49" event={"ID":"f0c069af-b064-47d5-ac44-b8ab45175489","Type":"ContainerDied","Data":"a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74"} Nov 03 01:20:46.492137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:46.492129 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ptwht-p8b49" Nov 03 01:20:46.492329 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:20:46.492175 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a65a88211d9942a03d9e707401e4b81be3203113b40ba5ca85c6383b867f8a74" Nov 03 01:21:02.149290 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:21:02.149250 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:21:32.150514 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:21:32.150477 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:22:02.150094 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:22:02.149955 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:22:29.959007 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:29.958971 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q"] Nov 03 01:22:29.959491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:29.959301 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f0c069af-b064-47d5-ac44-b8ab45175489" containerName="insights-gathering" Nov 03 01:22:29.959491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:29.959311 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c069af-b064-47d5-ac44-b8ab45175489" containerName="insights-gathering" Nov 03 01:22:29.959491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:29.959370 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c069af-b064-47d5-ac44-b8ab45175489" containerName="insights-gathering" Nov 03 01:22:29.962344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:29.962326 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.021404 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.021362 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q"] Nov 03 01:22:30.070569 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.070543 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-48lx4\"" Nov 03 01:22:30.070720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.070543 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 01:22:30.149359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149326 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149361 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvj8\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-kube-api-access-9vvj8\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149384 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149496 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149530 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149552 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149698 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f843b8a6-9841-44ac-99ca-a1c4577301b5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149751 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.149801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.149787 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250426 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250463 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250601 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f843b8a6-9841-44ac-99ca-a1c4577301b5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250644 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250678 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250697 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.250763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250720 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvj8\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-kube-api-access-9vvj8\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250956 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.250970 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.251042 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.251084 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.251107 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251317 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.251284 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.251405 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.251384 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f843b8a6-9841-44ac-99ca-a1c4577301b5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.252757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.252735 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.253281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.253265 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.316386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.316355 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.316697 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.316674 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvj8\" (UniqueName: \"kubernetes.io/projected/f843b8a6-9841-44ac-99ca-a1c4577301b5-kube-api-access-9vvj8\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q\" (UID: \"f843b8a6-9841-44ac-99ca-a1c4577301b5\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.575100 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.575015 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:30.996079 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:22:30.996036 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf843b8a6_9841_44ac_99ca_a1c4577301b5.slice/crio-61784bc9c071ff25208736673438a2ce645e0b7737bbff608a5c99a142edd39d WatchSource:0}: Error finding container 61784bc9c071ff25208736673438a2ce645e0b7737bbff608a5c99a142edd39d: Status 404 returned error can't find the container with id 61784bc9c071ff25208736673438a2ce645e0b7737bbff608a5c99a142edd39d Nov 03 01:22:30.997571 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:30.997530 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q"] Nov 03 01:22:31.909987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:31.909946 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" event={"ID":"f843b8a6-9841-44ac-99ca-a1c4577301b5","Type":"ContainerStarted","Data":"61784bc9c071ff25208736673438a2ce645e0b7737bbff608a5c99a142edd39d"} Nov 03 01:22:32.149200 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:22:32.149162 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:22:36.919676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:36.919644 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:36.919963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:36.919720 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:36.919963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:36.919751 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:22:37.941915 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:37.941871 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" event={"ID":"f843b8a6-9841-44ac-99ca-a1c4577301b5","Type":"ContainerStarted","Data":"3def9e66de7599d0143019b0479b597d48e207303b7ea770c5998eb8138fce5d"} Nov 03 01:22:38.170549 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:38.170485 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" podStartSLOduration=3.249727848 podStartE2EDuration="9.170470565s" totalImagesPullingTime="5.920742717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:38.118341082 +0000 UTC m=+293.697163669" watchObservedRunningTime="2026-11-03 01:22:38.170470565 +0000 UTC m=+293.749293221" Nov 03 01:22:38.575798 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:38.575759 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:38.580314 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:38.580289 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:38.946635 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:38.946607 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:38.947620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:38.947601 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q" Nov 03 01:22:44.604447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:44.604419 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:22:44.604876 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:22:44.604488 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:23:02.150438 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:23:02.150354 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:23:02.821935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.821896 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8"] Nov 03 01:23:02.828082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.828057 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.883118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.883084 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 01:23:02.883323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.883302 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 01:23:02.910908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.910869 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2kp7\" (UniqueName: \"kubernetes.io/projected/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-kube-api-access-z2kp7\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.911086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.910916 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.911086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.911035 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.911203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.911127 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.911203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.911180 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.911279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.911236 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:02.935013 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.934979 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 01:23:02.935013 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.935006 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rqx2g\"" Nov 03 01:23:02.935245 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.935123 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 01:23:02.935245 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.935203 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 01:23:02.936965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:02.936937 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8"] Nov 03 01:23:03.011994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.011955 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.011994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.011999 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.012238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.012020 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.012238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.012055 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2kp7\" (UniqueName: \"kubernetes.io/projected/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-kube-api-access-z2kp7\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.012238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.012091 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.012238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.012129 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.012774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.012746 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.014873 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.014851 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.015003 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.014969 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.015092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.015075 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.015129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.015095 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.082999 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.082923 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2kp7\" (UniqueName: \"kubernetes.io/projected/7d56eef0-4b7c-417c-8cdc-a723366cb5a6-kube-api-access-z2kp7\") pod \"opendatahub-operator-controller-manager-797bb7c459-vpvv8\" (UID: \"7d56eef0-4b7c-417c-8cdc-a723366cb5a6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.143598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.143545 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:03.679673 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:03.679637 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d56eef0_4b7c_417c_8cdc_a723366cb5a6.slice/crio-9865f4f07ea8f8a19da9b0cf6e77e56423c47dcf7594fcbdc56091a0f964429a WatchSource:0}: Error finding container 9865f4f07ea8f8a19da9b0cf6e77e56423c47dcf7594fcbdc56091a0f964429a: Status 404 returned error can't find the container with id 9865f4f07ea8f8a19da9b0cf6e77e56423c47dcf7594fcbdc56091a0f964429a Nov 03 01:23:03.680005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.679907 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8"] Nov 03 01:23:03.681413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:03.681394 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:23:04.048053 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:04.047964 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" event={"ID":"7d56eef0-4b7c-417c-8cdc-a723366cb5a6","Type":"ContainerStarted","Data":"9865f4f07ea8f8a19da9b0cf6e77e56423c47dcf7594fcbdc56091a0f964429a"} Nov 03 01:23:08.074465 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:08.074379 4992 generic.go:356] "Generic (PLEG): container finished" podID="7d56eef0-4b7c-417c-8cdc-a723366cb5a6" containerID="0930838808e2ba341417c177e2de4f03726949f37fc95be05f00e1b2e8ab3cc5" exitCode=0 Nov 03 01:23:08.074465 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:08.074420 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" event={"ID":"7d56eef0-4b7c-417c-8cdc-a723366cb5a6","Type":"ContainerDied","Data":"0930838808e2ba341417c177e2de4f03726949f37fc95be05f00e1b2e8ab3cc5"} Nov 03 01:23:09.081800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:09.081757 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" event={"ID":"7d56eef0-4b7c-417c-8cdc-a723366cb5a6","Type":"ContainerStarted","Data":"da4f86647dcba54fe843fb39cefccdb6ce7b96a15e5106f80fe0d928f4687a14"} Nov 03 01:23:09.082178 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:09.081881 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:09.334635 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:09.334496 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" podStartSLOduration=2.403223382 podStartE2EDuration="7.334476485s" totalImagesPullingTime="4.486469323s" totalInitContainerRuntime="444.78378ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:09.28228128 +0000 UTC m=+324.861103869" watchObservedRunningTime="2026-11-03 01:23:09.334476485 +0000 UTC m=+324.913299074" Nov 03 01:23:20.088676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.088640 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-vpvv8" Nov 03 01:23:20.513932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.513897 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb"] Nov 03 01:23:20.520376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.520351 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:20.578693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.578653 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 01:23:20.578843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.578689 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 01:23:20.578843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.578822 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rtj7b\"" Nov 03 01:23:20.581161 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.581123 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb"] Nov 03 01:23:20.665533 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.665494 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hplg\" (UniqueName: \"kubernetes.io/projected/8b089aeb-b68c-4a95-b72e-daa34a2d0886-kube-api-access-2hplg\") pod \"limitador-operator-controller-manager-5778bb4449-8f9fb\" (UID: \"8b089aeb-b68c-4a95-b72e-daa34a2d0886\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:20.766774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.766684 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hplg\" (UniqueName: \"kubernetes.io/projected/8b089aeb-b68c-4a95-b72e-daa34a2d0886-kube-api-access-2hplg\") pod \"limitador-operator-controller-manager-5778bb4449-8f9fb\" (UID: \"8b089aeb-b68c-4a95-b72e-daa34a2d0886\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:20.829699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.829663 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hplg\" (UniqueName: \"kubernetes.io/projected/8b089aeb-b68c-4a95-b72e-daa34a2d0886-kube-api-access-2hplg\") pod \"limitador-operator-controller-manager-5778bb4449-8f9fb\" (UID: \"8b089aeb-b68c-4a95-b72e-daa34a2d0886\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:20.833463 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:20.833437 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:21.205727 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:21.205690 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b089aeb_b68c_4a95_b72e_daa34a2d0886.slice/crio-e939b5a57e40a27b38a17f9ce1f2867db8ebec99266f2842f2d23710c0ab25ca WatchSource:0}: Error finding container e939b5a57e40a27b38a17f9ce1f2867db8ebec99266f2842f2d23710c0ab25ca: Status 404 returned error can't find the container with id e939b5a57e40a27b38a17f9ce1f2867db8ebec99266f2842f2d23710c0ab25ca Nov 03 01:23:21.208009 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:21.207983 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb"] Nov 03 01:23:22.134779 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:22.134736 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" event={"ID":"8b089aeb-b68c-4a95-b72e-daa34a2d0886","Type":"ContainerStarted","Data":"e939b5a57e40a27b38a17f9ce1f2867db8ebec99266f2842f2d23710c0ab25ca"} Nov 03 01:23:25.150170 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:25.150126 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" event={"ID":"8b089aeb-b68c-4a95-b72e-daa34a2d0886","Type":"ContainerStarted","Data":"6d7f295a6509414a67e1511e2f37fe5cbd693804f0a1dde835fdb28a37de9567"} Nov 03 01:23:25.150658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:25.150348 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:25.283923 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:25.283802 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" podStartSLOduration=1.522894006 podStartE2EDuration="5.283783497s" totalImagesPullingTime="3.760889491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:25.282765922 +0000 UTC m=+340.861588510" watchObservedRunningTime="2026-11-03 01:23:25.283783497 +0000 UTC m=+340.862606085" Nov 03 01:23:28.825886 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.825845 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6"] Nov 03 01:23:28.829312 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.829294 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:28.884059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.884031 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6"] Nov 03 01:23:28.886203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.886183 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 01:23:28.934717 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.934695 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 01:23:28.934860 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.934729 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ts4db\"" Nov 03 01:23:28.938205 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.938184 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:28.938298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:28.938283 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s54q\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-kube-api-access-2s54q\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.039594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.039545 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s54q\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-kube-api-access-2s54q\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.039779 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.039630 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.102963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.102885 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.102963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.102903 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s54q\" (UniqueName: \"kubernetes.io/projected/e7c56890-282a-4145-8af9-8aab1204eb9f-kube-api-access-2s54q\") pod \"cert-manager-cainjector-5c57bbd4fc-6l7w6\" (UID: \"e7c56890-282a-4145-8af9-8aab1204eb9f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.156900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.156867 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" Nov 03 01:23:29.523165 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:29.523127 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7c56890_282a_4145_8af9_8aab1204eb9f.slice/crio-b0655810dd85c125659c3da63b854b6ec4c12694cb8a459ff686b827481838a7 WatchSource:0}: Error finding container b0655810dd85c125659c3da63b854b6ec4c12694cb8a459ff686b827481838a7: Status 404 returned error can't find the container with id b0655810dd85c125659c3da63b854b6ec4c12694cb8a459ff686b827481838a7 Nov 03 01:23:29.525284 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:29.525260 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6"] Nov 03 01:23:30.174027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.173984 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" event={"ID":"e7c56890-282a-4145-8af9-8aab1204eb9f","Type":"ContainerStarted","Data":"b0655810dd85c125659c3da63b854b6ec4c12694cb8a459ff686b827481838a7"} Nov 03 01:23:30.701678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.701623 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx"] Nov 03 01:23:30.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.707833 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:30.837843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.837809 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zmzsn\"" Nov 03 01:23:30.839471 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.839448 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx"] Nov 03 01:23:30.857530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.857490 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffk9\" (UniqueName: \"kubernetes.io/projected/22284cad-4990-4345-ae90-b13bdc75e40a-kube-api-access-bffk9\") pod \"authorino-operator-7b7bb4bc77-nhqsx\" (UID: \"22284cad-4990-4345-ae90-b13bdc75e40a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:30.959061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:30.958963 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffk9\" (UniqueName: \"kubernetes.io/projected/22284cad-4990-4345-ae90-b13bdc75e40a-kube-api-access-bffk9\") pod \"authorino-operator-7b7bb4bc77-nhqsx\" (UID: \"22284cad-4990-4345-ae90-b13bdc75e40a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:31.024716 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:31.024679 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffk9\" (UniqueName: \"kubernetes.io/projected/22284cad-4990-4345-ae90-b13bdc75e40a-kube-api-access-bffk9\") pod \"authorino-operator-7b7bb4bc77-nhqsx\" (UID: \"22284cad-4990-4345-ae90-b13bdc75e40a\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:31.321126 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:31.321039 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:31.699069 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:31.699024 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx"] Nov 03 01:23:31.726934 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:31.726899 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22284cad_4990_4345_ae90_b13bdc75e40a.slice/crio-cb3bfeb451a744ad6d246c1a2fbb43220fd4c3879b371a00804b12b83e7507cb WatchSource:0}: Error finding container cb3bfeb451a744ad6d246c1a2fbb43220fd4c3879b371a00804b12b83e7507cb: Status 404 returned error can't find the container with id cb3bfeb451a744ad6d246c1a2fbb43220fd4c3879b371a00804b12b83e7507cb Nov 03 01:23:32.157645 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:23:32.157530 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:23:32.185695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:32.185653 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" event={"ID":"22284cad-4990-4345-ae90-b13bdc75e40a","Type":"ContainerStarted","Data":"cb3bfeb451a744ad6d246c1a2fbb43220fd4c3879b371a00804b12b83e7507cb"} Nov 03 01:23:35.170688 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.170652 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc"] Nov 03 01:23:35.174323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.174305 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194428 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbb0964-a624-4157-83c3-f59eee954cbf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194464 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqqj7\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-kube-api-access-tqqj7\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194540 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194697 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194645 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194697 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194672 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194781 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194822 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194913 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194900 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.194949 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.194929 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.202699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.202672 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" event={"ID":"e7c56890-282a-4145-8af9-8aab1204eb9f","Type":"ContainerStarted","Data":"760a818a9e36d1b12a7ceabe49b815ef3c9414a664bbf92466403d2e638a5345"} Nov 03 01:23:35.234666 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.234633 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9psjg\"" Nov 03 01:23:35.287463 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.287423 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc"] Nov 03 01:23:35.295380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295346 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295517 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295402 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295517 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295433 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295517 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295505 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295533 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295641 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbb0964-a624-4157-83c3-f59eee954cbf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295676 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqqj7\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-kube-api-access-tqqj7\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295870 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295712 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.295870 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295757 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.296412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.295976 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.296412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.296282 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.296412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.296362 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.296963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.296930 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.297083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.297045 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbb0964-a624-4157-83c3-f59eee954cbf-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.298820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.298801 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.298893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.298827 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.359378 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.359345 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqqj7\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-kube-api-access-tqqj7\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.362092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.362064 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbb0964-a624-4157-83c3-f59eee954cbf-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945svqcc\" (UID: \"8bbb0964-a624-4157-83c3-f59eee954cbf\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.424599 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.424466 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6l7w6" podStartSLOduration=2.802477196 podStartE2EDuration="7.424448359s" totalImagesPullingTime="4.621971163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:35.422688348 +0000 UTC m=+351.001510936" watchObservedRunningTime="2026-11-03 01:23:35.424448359 +0000 UTC m=+351.003270945" Nov 03 01:23:35.487383 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.487336 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:35.919774 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:35.919742 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbb0964_a624_4157_83c3_f59eee954cbf.slice/crio-1561b0cf22422eb0e7b0388db06f379f431d4e87b5f78100dfdcda68e740af5a WatchSource:0}: Error finding container 1561b0cf22422eb0e7b0388db06f379f431d4e87b5f78100dfdcda68e740af5a: Status 404 returned error can't find the container with id 1561b0cf22422eb0e7b0388db06f379f431d4e87b5f78100dfdcda68e740af5a Nov 03 01:23:35.922274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.922247 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc"] Nov 03 01:23:35.922697 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.922659 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:35.922787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.922759 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:35.922848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:35.922816 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:23:36.157480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.157403 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8f9fb" Nov 03 01:23:36.209249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.209215 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" event={"ID":"22284cad-4990-4345-ae90-b13bdc75e40a","Type":"ContainerStarted","Data":"c8b8735692c30469e0cf28d059a4196a5e2ef18503dbbe3db1b978fbbe7a640d"} Nov 03 01:23:36.209695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.209338 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:36.211025 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.210998 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" event={"ID":"8bbb0964-a624-4157-83c3-f59eee954cbf","Type":"ContainerStarted","Data":"45f2fdfeb01fa0e0b46e80d92c1fbfa32a8d2ccb0d3c0266a3009be0fa333b3e"} Nov 03 01:23:36.211160 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.211042 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" event={"ID":"8bbb0964-a624-4157-83c3-f59eee954cbf","Type":"ContainerStarted","Data":"1561b0cf22422eb0e7b0388db06f379f431d4e87b5f78100dfdcda68e740af5a"} Nov 03 01:23:36.424844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.424782 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" podStartSLOduration=2.288801195 podStartE2EDuration="6.424767634s" totalImagesPullingTime="4.135966439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:36.424042059 +0000 UTC m=+352.002864649" watchObservedRunningTime="2026-11-03 01:23:36.424767634 +0000 UTC m=+352.003590222" Nov 03 01:23:36.488400 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.488364 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:36.493141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.493118 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:36.548928 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.548876 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" podStartSLOduration=1.548861348 podStartE2EDuration="1.548861348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:36.547131994 +0000 UTC m=+352.125954580" watchObservedRunningTime="2026-11-03 01:23:36.548861348 +0000 UTC m=+352.127683935" Nov 03 01:23:36.826847 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.826769 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-dbqcs"] Nov 03 01:23:36.830453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.830436 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:36.885695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.885661 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xs48w\"" Nov 03 01:23:36.891087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.891058 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dbqcs"] Nov 03 01:23:36.909026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.908992 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4kr\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-kube-api-access-wj4kr\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:36.909210 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:36.909062 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-bound-sa-token\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.010329 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.010296 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4kr\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-kube-api-access-wj4kr\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.010519 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.010347 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-bound-sa-token\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.073995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.073952 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-bound-sa-token\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.074140 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.074067 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4kr\" (UniqueName: \"kubernetes.io/projected/37905007-04a2-4658-9a62-eb2ff5ec60e2-kube-api-access-wj4kr\") pod \"cert-manager-597b95cb84-dbqcs\" (UID: \"37905007-04a2-4658-9a62-eb2ff5ec60e2\") " pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.141735 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.141659 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-dbqcs" Nov 03 01:23:37.216373 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.216344 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:37.217385 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.217361 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945svqcc" Nov 03 01:23:37.515548 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:37.515507 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37905007_04a2_4658_9a62_eb2ff5ec60e2.slice/crio-c306fc0fb6834c8a4f664c81306ebd2b2665c03838fbe000c7c50c674cce2e77 WatchSource:0}: Error finding container c306fc0fb6834c8a4f664c81306ebd2b2665c03838fbe000c7c50c674cce2e77: Status 404 returned error can't find the container with id c306fc0fb6834c8a4f664c81306ebd2b2665c03838fbe000c7c50c674cce2e77 Nov 03 01:23:37.516844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:37.516822 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-dbqcs"] Nov 03 01:23:38.221231 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:38.221190 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dbqcs" event={"ID":"37905007-04a2-4658-9a62-eb2ff5ec60e2","Type":"ContainerStarted","Data":"27c73134596120dbe6161c1d0f36b505dbddde52e597f8cd3d85c05037f2a500"} Nov 03 01:23:38.221642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:38.221240 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-dbqcs" event={"ID":"37905007-04a2-4658-9a62-eb2ff5ec60e2","Type":"ContainerStarted","Data":"c306fc0fb6834c8a4f664c81306ebd2b2665c03838fbe000c7c50c674cce2e77"} Nov 03 01:23:38.348539 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:38.348489 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-dbqcs" podStartSLOduration=2.348476426 podStartE2EDuration="2.348476426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:38.348006936 +0000 UTC m=+353.926829523" watchObservedRunningTime="2026-11-03 01:23:38.348476426 +0000 UTC m=+353.927299014" Nov 03 01:23:41.573548 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.573509 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:23:41.582082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.582054 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:23:41.648927 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.648873 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psppq\" (UniqueName: \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\") pod \"authorino-56c6c9ddd5-fk8lt\" (UID: \"4a063f9c-a383-40a3-9c17-eb87ec5bc660\") " pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:23:41.663256 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.663223 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:23:41.722705 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.722667 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cm5nl\"" Nov 03 01:23:41.749950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.749912 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psppq\" (UniqueName: \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\") pod \"authorino-56c6c9ddd5-fk8lt\" (UID: \"4a063f9c-a383-40a3-9c17-eb87ec5bc660\") " pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:23:41.752143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.752113 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r"] Nov 03 01:23:41.756318 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.756301 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.785857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.785828 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r"] Nov 03 01:23:41.810709 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.810680 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 01:23:41.813272 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.813238 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psppq\" (UniqueName: \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\") pod \"authorino-56c6c9ddd5-fk8lt\" (UID: \"4a063f9c-a383-40a3-9c17-eb87ec5bc660\") " pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:23:41.822237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.822218 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 01:23:41.822353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.822318 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8vbrr\"" Nov 03 01:23:41.850878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.850789 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp4q\" (UniqueName: \"kubernetes.io/projected/9ffc6632-9a8c-46a2-8493-77016e28e030-kube-api-access-nqp4q\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.850878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.850849 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.851102 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.850956 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffc6632-9a8c-46a2-8493-77016e28e030-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.894627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.894596 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:23:41.952421 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.952387 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp4q\" (UniqueName: \"kubernetes.io/projected/9ffc6632-9a8c-46a2-8493-77016e28e030-kube-api-access-nqp4q\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.952590 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.952430 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.952590 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.952464 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffc6632-9a8c-46a2-8493-77016e28e030-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:41.952724 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:23:41.952631 4992 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 01:23:41.952724 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:23:41.952701 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert podName:9ffc6632-9a8c-46a2-8493-77016e28e030 nodeName:}" failed. No retries permitted until 2026-11-03 01:23:42.452682446 +0000 UTC m=+358.031505034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-pwl4r" (UID: "9ffc6632-9a8c-46a2-8493-77016e28e030") : secret "plugin-serving-cert" not found Nov 03 01:23:41.953247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:41.953225 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffc6632-9a8c-46a2-8493-77016e28e030-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:42.018055 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:42.018021 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp4q\" (UniqueName: \"kubernetes.io/projected/9ffc6632-9a8c-46a2-8493-77016e28e030-kube-api-access-nqp4q\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:42.262430 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:42.262384 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a063f9c_a383_40a3_9c17_eb87ec5bc660.slice/crio-f8e41ddc98fdb439bac2aeb8eaaac73726bcfbe72fe14c2da351d981f0637b16 WatchSource:0}: Error finding container f8e41ddc98fdb439bac2aeb8eaaac73726bcfbe72fe14c2da351d981f0637b16: Status 404 returned error can't find the container with id f8e41ddc98fdb439bac2aeb8eaaac73726bcfbe72fe14c2da351d981f0637b16 Nov 03 01:23:42.264066 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:42.264041 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:23:42.457165 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:42.457122 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:42.459689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:42.459669 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ffc6632-9a8c-46a2-8493-77016e28e030-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pwl4r\" (UID: \"9ffc6632-9a8c-46a2-8493-77016e28e030\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:42.667391 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:42.667356 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" Nov 03 01:23:43.035316 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:23:43.035285 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ffc6632_9a8c_46a2_8493_77016e28e030.slice/crio-12eedb0cd0b527d4721fcf008c25ca2ae984d773da4399eb6c131bf5a0876054 WatchSource:0}: Error finding container 12eedb0cd0b527d4721fcf008c25ca2ae984d773da4399eb6c131bf5a0876054: Status 404 returned error can't find the container with id 12eedb0cd0b527d4721fcf008c25ca2ae984d773da4399eb6c131bf5a0876054 Nov 03 01:23:43.037321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:43.037297 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r"] Nov 03 01:23:43.246929 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:43.246891 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" event={"ID":"4a063f9c-a383-40a3-9c17-eb87ec5bc660","Type":"ContainerStarted","Data":"f8e41ddc98fdb439bac2aeb8eaaac73726bcfbe72fe14c2da351d981f0637b16"} Nov 03 01:23:43.247998 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:43.247967 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" event={"ID":"9ffc6632-9a8c-46a2-8493-77016e28e030","Type":"ContainerStarted","Data":"12eedb0cd0b527d4721fcf008c25ca2ae984d773da4399eb6c131bf5a0876054"} Nov 03 01:23:45.260652 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:45.260607 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" event={"ID":"4a063f9c-a383-40a3-9c17-eb87ec5bc660","Type":"ContainerStarted","Data":"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6"} Nov 03 01:23:45.380708 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:45.380644 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" podStartSLOduration=1.735074573 podStartE2EDuration="4.380624092s" totalImagesPullingTime="2.645549519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:45.379721391 +0000 UTC m=+360.958543979" watchObservedRunningTime="2026-11-03 01:23:45.380624092 +0000 UTC m=+360.959446680" Nov 03 01:23:47.218410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:47.218377 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nhqsx" Nov 03 01:23:50.291828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:50.291745 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" event={"ID":"9ffc6632-9a8c-46a2-8493-77016e28e030","Type":"ContainerStarted","Data":"92885c3e148541331f2c7c5871885127dad0dc4d4d35a675a7204068f0a09032"} Nov 03 01:23:50.415651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:23:50.415594 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pwl4r" podStartSLOduration=2.448057275 podStartE2EDuration="9.415552515s" totalImagesPullingTime="6.96749524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:50.414945543 +0000 UTC m=+365.993768131" watchObservedRunningTime="2026-11-03 01:23:50.415552515 +0000 UTC m=+365.994375102" Nov 03 01:24:02.150682 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:24:02.150638 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:24:32.149759 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:24:32.149707 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:25:02.150437 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:02.150389 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:25:09.521777 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.521731 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:09.525331 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.525308 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.581295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.581259 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:09.620719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.620683 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.620909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.620735 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.620909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.620882 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46f4\" (UniqueName: \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.621037 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.620926 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.631550 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.631518 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:25:09.631704 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.631588 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:25:09.631704 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.631604 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 01:25:09.631704 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.631618 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:25:09.631704 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.631662 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\"" Nov 03 01:25:09.721925 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.721884 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46f4\" (UniqueName: \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.722095 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.721935 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.722095 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.721994 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.722095 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.722022 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.722719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.722694 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.724331 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.724306 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.724463 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.724444 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.787538 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.787456 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46f4\" (UniqueName: \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\") pod \"maas-api-5b7579b8fb-hnzsb\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:09.838364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:09.838317 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:10.208276 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:10.208247 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414a8e45_6900_4d0a_a708_68854f658053.slice/crio-4eca7a70224ee62087801c2cdf653af32e4999f801c75696501ced47df541bc4 WatchSource:0}: Error finding container 4eca7a70224ee62087801c2cdf653af32e4999f801c75696501ced47df541bc4: Status 404 returned error can't find the container with id 4eca7a70224ee62087801c2cdf653af32e4999f801c75696501ced47df541bc4 Nov 03 01:25:10.212101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.212076 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:10.649952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.649869 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-d6v2s"] Nov 03 01:25:10.656067 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.655430 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.663408 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.663377 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" event={"ID":"414a8e45-6900-4d0a-a708-68854f658053","Type":"ContainerStarted","Data":"4eca7a70224ee62087801c2cdf653af32e4999f801c75696501ced47df541bc4"} Nov 03 01:25:10.708169 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.708130 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-d6v2s"] Nov 03 01:25:10.710476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.710436 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 01:25:10.710737 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.710712 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 01:25:10.710853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.710779 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 01:25:10.710853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.710779 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-nlzw4\"" Nov 03 01:25:10.731940 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.731904 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f592be42-f99a-48e7-adb0-314512784195-cert\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.732072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.731951 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlm6\" (UniqueName: \"kubernetes.io/projected/f592be42-f99a-48e7-adb0-314512784195-kube-api-access-kwlm6\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.763612 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.763578 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:25:10.767211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.767187 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.821799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.821767 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:25:10.821985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.821917 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-stpkg\"" Nov 03 01:25:10.821985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.821944 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 01:25:10.833360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.833326 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctz9\" (UniqueName: \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.833504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.833410 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f592be42-f99a-48e7-adb0-314512784195-cert\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.833504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.833438 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlm6\" (UniqueName: \"kubernetes.io/projected/f592be42-f99a-48e7-adb0-314512784195-kube-api-access-kwlm6\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.833504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.833471 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.836197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.836172 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f592be42-f99a-48e7-adb0-314512784195-cert\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.884376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.884337 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6nfb8"] Nov 03 01:25:10.887871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.887852 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:10.897984 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.897958 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlm6\" (UniqueName: \"kubernetes.io/projected/f592be42-f99a-48e7-adb0-314512784195-kube-api-access-kwlm6\") pod \"kserve-controller-manager-776cc66d47-d6v2s\" (UID: \"f592be42-f99a-48e7-adb0-314512784195\") " pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.934407 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.934373 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.934645 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.934435 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctz9\" (UniqueName: \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.937268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.937244 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:10.943047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.943018 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 01:25:10.943175 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.943072 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-n4c5h\"" Nov 03 01:25:10.943437 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.943412 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6nfb8"] Nov 03 01:25:10.970224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.970193 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:10.999621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:10.999596 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctz9\" (UniqueName: \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\") pod \"llmisvc-controller-manager-869d5dcc5d-r9lmf\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:11.004154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.004126 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-vpzrm"] Nov 03 01:25:11.007985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.007962 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.035085 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.035059 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.035228 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.035207 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bl9t\" (UniqueName: \"kubernetes.io/projected/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-kube-api-access-4bl9t\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.063003 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.062955 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vpzrm"] Nov 03 01:25:11.066331 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.066306 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 01:25:11.066500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.066389 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 01:25:11.066589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.066523 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h2ffb\"" Nov 03 01:25:11.080671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.080651 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:11.136685 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.136655 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.136807 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.136786 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-cert\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.136873 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.136838 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg5t\" (UniqueName: \"kubernetes.io/projected/8972b343-b983-4b3c-879e-d4a6105ad81c-kube-api-access-lkg5t\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.136921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.136878 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bl9t\" (UniqueName: \"kubernetes.io/projected/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-kube-api-access-4bl9t\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.136980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.136921 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-metrics-certs\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.139651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.139628 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.200449 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.200381 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bl9t\" (UniqueName: \"kubernetes.io/projected/8b35e9d1-25de-4f51-af84-1b3f6a965f6c-kube-api-access-4bl9t\") pod \"model-serving-api-75bd89fdfc-6nfb8\" (UID: \"8b35e9d1-25de-4f51-af84-1b3f6a965f6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.202198 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.202171 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:11.238358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.238324 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-cert\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.238506 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.238375 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg5t\" (UniqueName: \"kubernetes.io/projected/8972b343-b983-4b3c-879e-d4a6105ad81c-kube-api-access-lkg5t\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.238506 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.238414 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-metrics-certs\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.241996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.241941 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-cert\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.241996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.241974 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8972b343-b983-4b3c-879e-d4a6105ad81c-metrics-certs\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.301827 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.301793 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg5t\" (UniqueName: \"kubernetes.io/projected/8972b343-b983-4b3c-879e-d4a6105ad81c-kube-api-access-lkg5t\") pod \"odh-model-controller-769875db56-vpzrm\" (UID: \"8972b343-b983-4b3c-879e-d4a6105ad81c\") " pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.324450 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.324420 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:11.343743 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:11.343709 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf592be42_f99a_48e7_adb0_314512784195.slice/crio-0a8edf4934c39046b32b001a27cda38dbfac27cb676b2341d1849e3531645200 WatchSource:0}: Error finding container 0a8edf4934c39046b32b001a27cda38dbfac27cb676b2341d1849e3531645200: Status 404 returned error can't find the container with id 0a8edf4934c39046b32b001a27cda38dbfac27cb676b2341d1849e3531645200 Nov 03 01:25:11.345745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.345722 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-d6v2s"] Nov 03 01:25:11.450363 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:11.450325 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod102eadc4_5f4f_408a_9e87_f7f5c98c8780.slice/crio-ba34cb164f095dd3b2cfe04b676a08769c7b8b7d52d6365a202f6e1813c98386 WatchSource:0}: Error finding container ba34cb164f095dd3b2cfe04b676a08769c7b8b7d52d6365a202f6e1813c98386: Status 404 returned error can't find the container with id ba34cb164f095dd3b2cfe04b676a08769c7b8b7d52d6365a202f6e1813c98386 Nov 03 01:25:11.451801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.451776 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:25:11.572162 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:11.572127 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b35e9d1_25de_4f51_af84_1b3f6a965f6c.slice/crio-d1e542ca1d2261d43894b1e32f2093ac8537350fca182c37e8540638b772025b WatchSource:0}: Error finding container d1e542ca1d2261d43894b1e32f2093ac8537350fca182c37e8540638b772025b: Status 404 returned error can't find the container with id d1e542ca1d2261d43894b1e32f2093ac8537350fca182c37e8540638b772025b Nov 03 01:25:11.573999 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.573971 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6nfb8"] Nov 03 01:25:11.669799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.669763 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" event={"ID":"8b35e9d1-25de-4f51-af84-1b3f6a965f6c","Type":"ContainerStarted","Data":"d1e542ca1d2261d43894b1e32f2093ac8537350fca182c37e8540638b772025b"} Nov 03 01:25:11.671045 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.671012 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" event={"ID":"102eadc4-5f4f-408a-9e87-f7f5c98c8780","Type":"ContainerStarted","Data":"ba34cb164f095dd3b2cfe04b676a08769c7b8b7d52d6365a202f6e1813c98386"} Nov 03 01:25:11.672256 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.672236 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerStarted","Data":"0a8edf4934c39046b32b001a27cda38dbfac27cb676b2341d1849e3531645200"} Nov 03 01:25:11.696748 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:11.696703 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8972b343_b983_4b3c_879e_d4a6105ad81c.slice/crio-698b626882387fab2f7aeea3d32e4bdea14aebefe5867efa347fd0b2d2382f15 WatchSource:0}: Error finding container 698b626882387fab2f7aeea3d32e4bdea14aebefe5867efa347fd0b2d2382f15: Status 404 returned error can't find the container with id 698b626882387fab2f7aeea3d32e4bdea14aebefe5867efa347fd0b2d2382f15 Nov 03 01:25:11.698466 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:11.698438 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vpzrm"] Nov 03 01:25:12.681962 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:12.681922 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vpzrm" event={"ID":"8972b343-b983-4b3c-879e-d4a6105ad81c","Type":"ContainerStarted","Data":"698b626882387fab2f7aeea3d32e4bdea14aebefe5867efa347fd0b2d2382f15"} Nov 03 01:25:15.721345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:15.720324 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" event={"ID":"8b35e9d1-25de-4f51-af84-1b3f6a965f6c","Type":"ContainerStarted","Data":"20fe209f6d867a47f2d95f202d35ba4ad536a6fde793adaed8db2b96e0223570"} Nov 03 01:25:15.721345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:15.721287 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:15.842929 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:15.842806 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" podStartSLOduration=1.874538458 podStartE2EDuration="5.842788127s" totalImagesPullingTime="3.968249669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:15.841087437 +0000 UTC m=+451.419910025" watchObservedRunningTime="2026-11-03 01:25:15.842788127 +0000 UTC m=+451.421610714" Nov 03 01:25:16.730986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:16.730924 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerStarted","Data":"d7a6c1215a9b6407218b508a94b18a12b053bf8f3b45ec5fc1e2481c9a3aba32"} Nov 03 01:25:16.731335 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:16.731280 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:16.852112 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:16.852053 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" podStartSLOduration=1.748168176 podStartE2EDuration="6.852040777s" totalImagesPullingTime="5.103872601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:16.850945638 +0000 UTC m=+452.429768225" watchObservedRunningTime="2026-11-03 01:25:16.852040777 +0000 UTC m=+452.430863363" Nov 03 01:25:17.737971 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.737874 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" event={"ID":"414a8e45-6900-4d0a-a708-68854f658053","Type":"ContainerStarted","Data":"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d"} Nov 03 01:25:17.737971 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.737951 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:17.739966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.739940 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vpzrm" event={"ID":"8972b343-b983-4b3c-879e-d4a6105ad81c","Type":"ContainerStarted","Data":"1f50aeaeb36f031ba89ef42e39ca2d8703aa4337a4897a6802b0007b1a65eba4"} Nov 03 01:25:17.740343 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.740309 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:17.861304 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.861253 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" podStartSLOduration=1.629023606 podStartE2EDuration="8.861231656s" totalImagesPullingTime="7.23220805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:17.860690175 +0000 UTC m=+453.439512763" watchObservedRunningTime="2026-11-03 01:25:17.861231656 +0000 UTC m=+453.440054233" Nov 03 01:25:17.981693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:17.981642 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-vpzrm" podStartSLOduration=2.971693619 podStartE2EDuration="7.981623768s" totalImagesPullingTime="5.009930149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:17.980081174 +0000 UTC m=+453.558903762" watchObservedRunningTime="2026-11-03 01:25:17.981623768 +0000 UTC m=+453.560446356" Nov 03 01:25:18.747250 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:18.747206 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" event={"ID":"102eadc4-5f4f-408a-9e87-f7f5c98c8780","Type":"ContainerStarted","Data":"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8"} Nov 03 01:25:18.872762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:18.872685 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" podStartSLOduration=2.390382364 podStartE2EDuration="8.872664735s" totalImagesPullingTime="6.482282371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:18.870947321 +0000 UTC m=+454.449769908" watchObservedRunningTime="2026-11-03 01:25:18.872664735 +0000 UTC m=+454.451487320" Nov 03 01:25:19.754380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:19.754348 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:25:23.753077 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:23.753048 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:24.609659 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.609626 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:24.613355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.613333 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:24.665812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.665775 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:24.768384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.768348 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgr49\" (UniqueName: \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\") pod \"authorino-6f6944bcdd-tlzxx\" (UID: \"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:24.869137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.869051 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgr49\" (UniqueName: \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\") pod \"authorino-6f6944bcdd-tlzxx\" (UID: \"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:24.932207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.932177 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgr49\" (UniqueName: \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\") pod \"authorino-6f6944bcdd-tlzxx\" (UID: \"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b\") " pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:24.982208 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.982173 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:24.982386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:24.982375 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:25.009979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.009951 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:25.014656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.014630 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:25.067771 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.067738 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:25.172294 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.172261 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56q4l\" (UniqueName: \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\") pod \"authorino-7c6f9f748-zp29w\" (UID: \"3590313b-92cb-4cba-9975-046fb069b379\") " pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:25.272875 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.272838 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56q4l\" (UniqueName: \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\") pod \"authorino-7c6f9f748-zp29w\" (UID: \"3590313b-92cb-4cba-9975-046fb069b379\") " pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:25.335833 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.335767 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56q4l\" (UniqueName: \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\") pod \"authorino-7c6f9f748-zp29w\" (UID: \"3590313b-92cb-4cba-9975-046fb069b379\") " pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:25.351964 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:25.351932 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e72ef8_0c6d_42fc_8cba_20aeca70ee2b.slice/crio-e9442462217152886cff5366dd7d83555972218890585916fdd8a9a3a1edd0e7 WatchSource:0}: Error finding container e9442462217152886cff5366dd7d83555972218890585916fdd8a9a3a1edd0e7: Status 404 returned error can't find the container with id e9442462217152886cff5366dd7d83555972218890585916fdd8a9a3a1edd0e7 Nov 03 01:25:25.354008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.353985 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:25.450327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.450241 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:25.450478 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.450459 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:25.481526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.481496 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:25:25.486428 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.486400 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.537680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.537645 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:25:25.540740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.540715 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 01:25:25.576187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.576148 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.576338 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.576296 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dsz\" (UniqueName: \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.677469 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.677427 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dsz\" (UniqueName: \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.677670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.677487 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.679920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.679887 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.740207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.740127 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dsz\" (UniqueName: \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\") pod \"authorino-57c96bc49d-gghrk\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.788693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.788654 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" event={"ID":"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b","Type":"ContainerStarted","Data":"e9442462217152886cff5366dd7d83555972218890585916fdd8a9a3a1edd0e7"} Nov 03 01:25:25.801911 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.801879 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:25:25.824135 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:25.824102 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3590313b_92cb_4cba_9975_046fb069b379.slice/crio-38befec5b3e9e4e9c00647821da640f960d2a6d8b2ecdf9b08afc59b27e1b6da WatchSource:0}: Error finding container 38befec5b3e9e4e9c00647821da640f960d2a6d8b2ecdf9b08afc59b27e1b6da: Status 404 returned error can't find the container with id 38befec5b3e9e4e9c00647821da640f960d2a6d8b2ecdf9b08afc59b27e1b6da Nov 03 01:25:25.826342 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:25.826320 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:26.171932 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:25:26.171901 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd497c97c_628a_4bb7_a663_bc97d1aae1d2.slice/crio-41ef4dc83640b59fa50852e17f96ef6c4926d41dd031059dfd41506f957ba087 WatchSource:0}: Error finding container 41ef4dc83640b59fa50852e17f96ef6c4926d41dd031059dfd41506f957ba087: Status 404 returned error can't find the container with id 41ef4dc83640b59fa50852e17f96ef6c4926d41dd031059dfd41506f957ba087 Nov 03 01:25:26.173762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:26.173735 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:25:26.794102 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:26.794070 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6f9f748-zp29w" event={"ID":"3590313b-92cb-4cba-9975-046fb069b379","Type":"ContainerStarted","Data":"38befec5b3e9e4e9c00647821da640f960d2a6d8b2ecdf9b08afc59b27e1b6da"} Nov 03 01:25:26.795184 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:26.795161 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c96bc49d-gghrk" event={"ID":"d497c97c-628a-4bb7-a663-bc97d1aae1d2","Type":"ContainerStarted","Data":"41ef4dc83640b59fa50852e17f96ef6c4926d41dd031059dfd41506f957ba087"} Nov 03 01:25:27.744501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.744471 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6nfb8" Nov 03 01:25:27.802292 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.802189 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6f9f748-zp29w" event={"ID":"3590313b-92cb-4cba-9975-046fb069b379","Type":"ContainerStarted","Data":"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984"} Nov 03 01:25:27.802292 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.802216 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7c6f9f748-zp29w" podUID="3590313b-92cb-4cba-9975-046fb069b379" containerName="authorino" containerID="cri-o://34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984" gracePeriod=30 Nov 03 01:25:27.806214 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.806161 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c96bc49d-gghrk" event={"ID":"d497c97c-628a-4bb7-a663-bc97d1aae1d2","Type":"ContainerStarted","Data":"b3c8074dad3936e7fc6dfc1f2ba4503087f4c3343de197f91cab6cf3780458ed"} Nov 03 01:25:27.809492 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.809468 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" event={"ID":"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b","Type":"ContainerStarted","Data":"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36"} Nov 03 01:25:27.809657 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.809627 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" podUID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" containerName="authorino" containerID="cri-o://65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36" gracePeriod=30 Nov 03 01:25:27.984694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:27.983833 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c6f9f748-zp29w" podStartSLOduration=2.530682625 podStartE2EDuration="3.983811673s" totalImagesPullingTime="1.453129048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:27.983694686 +0000 UTC m=+463.562517274" watchObservedRunningTime="2026-11-03 01:25:27.983811673 +0000 UTC m=+463.562634263" Nov 03 01:25:28.077943 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.077916 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:28.086507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.086479 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:28.210102 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.210066 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\" (UniqueName: \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\") pod \"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b\" (UID: \"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b\") " Nov 03 01:25:28.210297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.210155 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\" (UniqueName: \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\") pod \"3590313b-92cb-4cba-9975-046fb069b379\" (UID: \"3590313b-92cb-4cba-9975-046fb069b379\") " Nov 03 01:25:28.212528 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.212497 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l" pod "3590313b-92cb-4cba-9975-046fb069b379" (UID: "3590313b-92cb-4cba-9975-046fb069b379"). InnerVolumeSpecName "kube-api-access-56q4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:28.212674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.212545 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49" pod "44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" (UID: "44e72ef8-0c6d-42fc-8cba-20aeca70ee2b"). InnerVolumeSpecName "kube-api-access-sgr49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:28.231990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.230635 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57c96bc49d-gghrk" podStartSLOduration=1.891191318 podStartE2EDuration="3.230616155s" totalImagesPullingTime="1.339424837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:28.229801075 +0000 UTC m=+463.808623658" watchObservedRunningTime="2026-11-03 01:25:28.230616155 +0000 UTC m=+463.809438740" Nov 03 01:25:28.232340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.232285 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" podStartSLOduration=2.352768426 podStartE2EDuration="4.232268479s" totalImagesPullingTime="1.879500053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:28.107965123 +0000 UTC m=+463.686787710" watchObservedRunningTime="2026-11-03 01:25:28.232268479 +0000 UTC m=+463.811091149" Nov 03 01:25:28.274627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.259607 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:25:28.274627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.259873 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" podUID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" containerName="authorino" containerID="cri-o://ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6" gracePeriod=30 Nov 03 01:25:28.311004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.310970 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56q4l\" (UniqueName: \"kubernetes.io/projected/3590313b-92cb-4cba-9975-046fb069b379-kube-api-access-56q4l\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:28.311004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.311001 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgr49\" (UniqueName: \"kubernetes.io/projected/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b-kube-api-access-sgr49\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:28.547925 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.547890 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:25:28.713321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.713288 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\" (UniqueName: \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\") pod \"4a063f9c-a383-40a3-9c17-eb87ec5bc660\" (UID: \"4a063f9c-a383-40a3-9c17-eb87ec5bc660\") " Nov 03 01:25:28.715646 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.715611 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq" pod "4a063f9c-a383-40a3-9c17-eb87ec5bc660" (UID: "4a063f9c-a383-40a3-9c17-eb87ec5bc660"). InnerVolumeSpecName "kube-api-access-psppq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:28.750112 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.750083 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-vpzrm" Nov 03 01:25:28.814089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.814056 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psppq\" (UniqueName: \"kubernetes.io/projected/4a063f9c-a383-40a3-9c17-eb87ec5bc660-kube-api-access-psppq\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:28.816082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.816051 4992 generic.go:356] "Generic (PLEG): container finished" podID="3590313b-92cb-4cba-9975-046fb069b379" containerID="34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984" exitCode=0 Nov 03 01:25:28.816196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.816105 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c6f9f748-zp29w" Nov 03 01:25:28.816196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.816125 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6f9f748-zp29w" event={"ID":"3590313b-92cb-4cba-9975-046fb069b379","Type":"ContainerDied","Data":"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984"} Nov 03 01:25:28.816196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.816165 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c6f9f748-zp29w" event={"ID":"3590313b-92cb-4cba-9975-046fb069b379","Type":"ContainerDied","Data":"38befec5b3e9e4e9c00647821da640f960d2a6d8b2ecdf9b08afc59b27e1b6da"} Nov 03 01:25:28.816196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.816184 4992 scope.go:122] "RemoveContainer" containerID="34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984" Nov 03 01:25:28.817596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.817573 4992 generic.go:356] "Generic (PLEG): container finished" podID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" containerID="65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36" exitCode=0 Nov 03 01:25:28.817791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.817604 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" event={"ID":"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b","Type":"ContainerDied","Data":"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36"} Nov 03 01:25:28.818241 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.817813 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" event={"ID":"44e72ef8-0c6d-42fc-8cba-20aeca70ee2b","Type":"ContainerDied","Data":"e9442462217152886cff5366dd7d83555972218890585916fdd8a9a3a1edd0e7"} Nov 03 01:25:28.818241 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.817614 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-tlzxx" Nov 03 01:25:28.819528 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.819505 4992 generic.go:356] "Generic (PLEG): container finished" podID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" containerID="ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6" exitCode=0 Nov 03 01:25:28.819660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.819590 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" Nov 03 01:25:28.819660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.819599 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" event={"ID":"4a063f9c-a383-40a3-9c17-eb87ec5bc660","Type":"ContainerDied","Data":"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6"} Nov 03 01:25:28.819660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.819637 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fk8lt" event={"ID":"4a063f9c-a383-40a3-9c17-eb87ec5bc660","Type":"ContainerDied","Data":"f8e41ddc98fdb439bac2aeb8eaaac73726bcfbe72fe14c2da351d981f0637b16"} Nov 03 01:25:28.825782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.825764 4992 scope.go:122] "RemoveContainer" containerID="34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984" Nov 03 01:25:28.826105 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:28.826078 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984\": container with ID starting with 34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984 not found: ID does not exist" containerID="34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984" Nov 03 01:25:28.826218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.826120 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984"} err="failed to get container status \"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984\": rpc error: code = NotFound desc = could not find container \"34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984\": container with ID starting with 34f6ee8f65fbecb22454f39abae1c3c61bc2807ae923c32696e79f35ad7cd984 not found: ID does not exist" Nov 03 01:25:28.826218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.826143 4992 scope.go:122] "RemoveContainer" containerID="65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36" Nov 03 01:25:28.834024 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.834009 4992 scope.go:122] "RemoveContainer" containerID="65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36" Nov 03 01:25:28.834279 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:28.834261 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36\": container with ID starting with 65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36 not found: ID does not exist" containerID="65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36" Nov 03 01:25:28.834327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.834285 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36"} err="failed to get container status \"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36\": rpc error: code = NotFound desc = could not find container \"65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36\": container with ID starting with 65407e8e6209a8fb713768c75b0e8169e216357628b78fdfbb8192abd9525a36 not found: ID does not exist" Nov 03 01:25:28.834327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.834300 4992 scope.go:122] "RemoveContainer" containerID="ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6" Nov 03 01:25:28.841875 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.841860 4992 scope.go:122] "RemoveContainer" containerID="ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6" Nov 03 01:25:28.842133 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:28.842108 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6\": container with ID starting with ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6 not found: ID does not exist" containerID="ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6" Nov 03 01:25:28.842177 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.842135 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6"} err="failed to get container status \"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6\": rpc error: code = NotFound desc = could not find container \"ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6\": container with ID starting with ad4fc9054adcc7f7be8b90c92778a757ea962d7b2d41441dca65fbc93f052ec6 not found: ID does not exist" Nov 03 01:25:28.984604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.984518 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:28.992269 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:28.992238 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-tlzxx"] Nov 03 01:25:29.105007 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:29.104972 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:25:29.108844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:29.108819 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fk8lt"] Nov 03 01:25:29.221429 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:29.221397 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:29.225462 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:29.225436 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c6f9f748-zp29w"] Nov 03 01:25:30.593705 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:30.593669 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3590313b-92cb-4cba-9975-046fb069b379" path="/var/lib/kubelet/pods/3590313b-92cb-4cba-9975-046fb069b379/volumes" Nov 03 01:25:30.594103 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:30.594024 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" path="/var/lib/kubelet/pods/44e72ef8-0c6d-42fc-8cba-20aeca70ee2b/volumes" Nov 03 01:25:30.594318 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:30.594304 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" path="/var/lib/kubelet/pods/4a063f9c-a383-40a3-9c17-eb87ec5bc660/volumes" Nov 03 01:25:32.151347 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:32.151301 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:25:34.907600 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:34.907538 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:34.908052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:34.907788 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" podUID="414a8e45-6900-4d0a-a708-68854f658053" containerName="maas-api" containerID="cri-o://22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d" gracePeriod=30 Nov 03 01:25:35.164210 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.164188 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:35.261796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.261741 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\") pod \"414a8e45-6900-4d0a-a708-68854f658053\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " Nov 03 01:25:35.261980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.261814 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\") pod \"414a8e45-6900-4d0a-a708-68854f658053\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " Nov 03 01:25:35.261980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.261904 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\" (UniqueName: \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\") pod \"414a8e45-6900-4d0a-a708-68854f658053\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " Nov 03 01:25:35.262105 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.261980 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\") pod \"414a8e45-6900-4d0a-a708-68854f658053\" (UID: \"414a8e45-6900-4d0a-a708-68854f658053\") " Nov 03 01:25:35.262355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.262328 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca" pod "414a8e45-6900-4d0a-a708-68854f658053" (UID: "414a8e45-6900-4d0a-a708-68854f658053"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:35.264164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.264138 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls" pod "414a8e45-6900-4d0a-a708-68854f658053" (UID: "414a8e45-6900-4d0a-a708-68854f658053"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:35.264164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.264154 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4" pod "414a8e45-6900-4d0a-a708-68854f658053" (UID: "414a8e45-6900-4d0a-a708-68854f658053"). InnerVolumeSpecName "kube-api-access-l46f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:35.264306 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.264155 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs" pod "414a8e45-6900-4d0a-a708-68854f658053" (UID: "414a8e45-6900-4d0a-a708-68854f658053"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:35.363543 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.363498 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:35.363543 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.363536 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/414a8e45-6900-4d0a-a708-68854f658053-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:35.363543 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.363550 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46f4\" (UniqueName: \"kubernetes.io/projected/414a8e45-6900-4d0a-a708-68854f658053-kube-api-access-l46f4\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:35.363813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.363595 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414a8e45-6900-4d0a-a708-68854f658053-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:25:35.856513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.856471 4992 generic.go:356] "Generic (PLEG): container finished" podID="414a8e45-6900-4d0a-a708-68854f658053" containerID="22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d" exitCode=0 Nov 03 01:25:35.856718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.856523 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" Nov 03 01:25:35.856718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.856571 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" event={"ID":"414a8e45-6900-4d0a-a708-68854f658053","Type":"ContainerDied","Data":"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d"} Nov 03 01:25:35.856718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.856633 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb" event={"ID":"414a8e45-6900-4d0a-a708-68854f658053","Type":"ContainerDied","Data":"4eca7a70224ee62087801c2cdf653af32e4999f801c75696501ced47df541bc4"} Nov 03 01:25:35.856718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.856657 4992 scope.go:122] "RemoveContainer" containerID="22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d" Nov 03 01:25:35.866633 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.866613 4992 scope.go:122] "RemoveContainer" containerID="22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d" Nov 03 01:25:35.866922 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:25:35.866904 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d\": container with ID starting with 22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d not found: ID does not exist" containerID="22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d" Nov 03 01:25:35.866965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:35.866932 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d"} err="failed to get container status \"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d\": rpc error: code = NotFound desc = could not find container \"22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d\": container with ID starting with 22afda14a6a8bd7debdbc787017a53ceb85ed87ce160b163cae9455ac0bae95d not found: ID does not exist" Nov 03 01:25:36.027116 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:36.027083 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:36.078836 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:36.078804 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b7579b8fb-hnzsb"] Nov 03 01:25:36.598345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:36.598312 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414a8e45-6900-4d0a-a708-68854f658053" path="/var/lib/kubelet/pods/414a8e45-6900-4d0a-a708-68854f658053/volumes" Nov 03 01:25:47.745695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:47.745663 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:25:50.761707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:25:50.761671 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:26:02.149831 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:26:02.149742 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:26:29.813987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.813950 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2"] Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814353 4992 cpu_manager.go:414] "containerMap: removing container" podUID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" containerName="authorino" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814364 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" containerName="authorino" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814384 4992 cpu_manager.go:414] "containerMap: removing container" podUID="414a8e45-6900-4d0a-a708-68854f658053" containerName="maas-api" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814390 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414a8e45-6900-4d0a-a708-68854f658053" containerName="maas-api" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814396 4992 cpu_manager.go:414] "containerMap: removing container" podUID="3590313b-92cb-4cba-9975-046fb069b379" containerName="authorino" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814401 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3590313b-92cb-4cba-9975-046fb069b379" containerName="authorino" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814410 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" containerName="authorino" Nov 03 01:26:29.814480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814415 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" containerName="authorino" Nov 03 01:26:29.814787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814498 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e72ef8-0c6d-42fc-8cba-20aeca70ee2b" containerName="authorino" Nov 03 01:26:29.814787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814508 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="414a8e45-6900-4d0a-a708-68854f658053" containerName="maas-api" Nov 03 01:26:29.814787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814513 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="3590313b-92cb-4cba-9975-046fb069b379" containerName="authorino" Nov 03 01:26:29.814787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.814518 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a063f9c-a383-40a3-9c17-eb87ec5bc660" containerName="authorino" Nov 03 01:26:29.817725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.817701 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:29.849984 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.849950 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289f89f9-3d93-430e-9df8-0600cdd9b040-cert\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:29.850146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.849991 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8xzz\" (UniqueName: \"kubernetes.io/projected/289f89f9-3d93-430e-9df8-0600cdd9b040-kube-api-access-x8xzz\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:29.888527 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.888487 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2"] Nov 03 01:26:29.951447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.951401 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289f89f9-3d93-430e-9df8-0600cdd9b040-cert\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:29.951657 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.951465 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8xzz\" (UniqueName: \"kubernetes.io/projected/289f89f9-3d93-430e-9df8-0600cdd9b040-kube-api-access-x8xzz\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:29.953937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:29.953910 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/289f89f9-3d93-430e-9df8-0600cdd9b040-cert\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:30.015094 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:30.015064 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8xzz\" (UniqueName: \"kubernetes.io/projected/289f89f9-3d93-430e-9df8-0600cdd9b040-kube-api-access-x8xzz\") pod \"llmisvc-controller-manager-6466c554d8-sjhd2\" (UID: \"289f89f9-3d93-430e-9df8-0600cdd9b040\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:30.130264 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:30.130183 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:30.502426 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:26:30.502397 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289f89f9_3d93_430e_9df8_0600cdd9b040.slice/crio-396d699d60487a04de8e3fdb2126e52df30836dba61297d1a43e9daa97e975e6 WatchSource:0}: Error finding container 396d699d60487a04de8e3fdb2126e52df30836dba61297d1a43e9daa97e975e6: Status 404 returned error can't find the container with id 396d699d60487a04de8e3fdb2126e52df30836dba61297d1a43e9daa97e975e6 Nov 03 01:26:30.503791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:30.503762 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2"] Nov 03 01:26:31.123584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:31.123529 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" event={"ID":"289f89f9-3d93-430e-9df8-0600cdd9b040","Type":"ContainerStarted","Data":"396d699d60487a04de8e3fdb2126e52df30836dba61297d1a43e9daa97e975e6"} Nov 03 01:26:32.130459 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:32.130425 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" event={"ID":"289f89f9-3d93-430e-9df8-0600cdd9b040","Type":"ContainerStarted","Data":"884a1dab4814ee70448c61878daf5dbfb955f11c248c2fa1b03b20097b312a2a"} Nov 03 01:26:32.130932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:32.130511 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:26:32.150224 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:26:32.150190 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:26:32.251272 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:26:32.251219 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" podStartSLOduration=2.542858204 podStartE2EDuration="3.251203653s" totalImagesPullingTime="708.345449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:32.250623805 +0000 UTC m=+527.829446393" watchObservedRunningTime="2026-11-03 01:26:32.251203653 +0000 UTC m=+527.830026239" Nov 03 01:27:02.149529 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:27:02.149482 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:27:03.138385 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.138358 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sjhd2" Nov 03 01:27:03.311729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.311687 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:27:03.312208 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.311956 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" podUID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" containerName="manager" containerID="cri-o://111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8" gracePeriod=30 Nov 03 01:27:03.554309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.554282 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:27:03.660316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.660278 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\" (UniqueName: \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\") pod \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " Nov 03 01:27:03.660477 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.660373 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\" (UniqueName: \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\") pod \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\" (UID: \"102eadc4-5f4f-408a-9e87-f7f5c98c8780\") " Nov 03 01:27:03.662505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.662471 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert" pod "102eadc4-5f4f-408a-9e87-f7f5c98c8780" (UID: "102eadc4-5f4f-408a-9e87-f7f5c98c8780"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:03.662628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.662541 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9" pod "102eadc4-5f4f-408a-9e87-f7f5c98c8780" (UID: "102eadc4-5f4f-408a-9e87-f7f5c98c8780"). InnerVolumeSpecName "kube-api-access-vctz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:03.761320 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.761230 4992 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102eadc4-5f4f-408a-9e87-f7f5c98c8780-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:03.761320 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:03.761267 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctz9\" (UniqueName: \"kubernetes.io/projected/102eadc4-5f4f-408a-9e87-f7f5c98c8780-kube-api-access-vctz9\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:27:04.288916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.288883 4992 generic.go:356] "Generic (PLEG): container finished" podID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" containerID="111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8" exitCode=0 Nov 03 01:27:04.289086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.288939 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" Nov 03 01:27:04.289086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.288965 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" event={"ID":"102eadc4-5f4f-408a-9e87-f7f5c98c8780","Type":"ContainerDied","Data":"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8"} Nov 03 01:27:04.289086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.289020 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf" event={"ID":"102eadc4-5f4f-408a-9e87-f7f5c98c8780","Type":"ContainerDied","Data":"ba34cb164f095dd3b2cfe04b676a08769c7b8b7d52d6365a202f6e1813c98386"} Nov 03 01:27:04.289086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.289053 4992 scope.go:122] "RemoveContainer" containerID="111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8" Nov 03 01:27:04.298070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.298049 4992 scope.go:122] "RemoveContainer" containerID="111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8" Nov 03 01:27:04.298361 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:27:04.298339 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8\": container with ID starting with 111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8 not found: ID does not exist" containerID="111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8" Nov 03 01:27:04.298466 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.298373 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8"} err="failed to get container status \"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8\": rpc error: code = NotFound desc = could not find container \"111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8\": container with ID starting with 111cdb154040f690d2ef8240ad90605877383b479386e151f16152b83b08bcd8 not found: ID does not exist" Nov 03 01:27:04.459462 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.459430 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:27:04.463218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.463192 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-r9lmf"] Nov 03 01:27:04.593013 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:04.592938 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" path="/var/lib/kubelet/pods/102eadc4-5f4f-408a-9e87-f7f5c98c8780/volumes" Nov 03 01:27:32.149128 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:27:32.149091 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:27:44.633200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:44.633162 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:27:44.634024 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:27:44.633999 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:28:02.152550 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:28:02.152506 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:28:17.369094 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.369058 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:28:17.369611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.369466 4992 cpu_manager.go:414] "containerMap: removing container" podUID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" containerName="manager" Nov 03 01:28:17.369611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.369477 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" containerName="manager" Nov 03 01:28:17.369611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.369591 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="102eadc4-5f4f-408a-9e87-f7f5c98c8780" containerName="manager" Nov 03 01:28:17.372605 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.372586 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.404693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.404658 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.404826 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.404705 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.404867 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.404826 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.404901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.404870 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzgw\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.404938 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.404901 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.405026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.405006 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.405074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.405062 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.405108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.405086 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.405143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.405107 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.479015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.478986 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\"" Nov 03 01:28:17.481144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.481112 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:28:17.506364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506330 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506379 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506404 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506435 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506471 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506500 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzgw\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506528 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506641 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506679 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.506927 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.506807 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.507288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.507267 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.507456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.507297 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.507456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.507370 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.507676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.507658 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.509077 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.509050 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.509283 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.509248 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.569321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.569288 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.569582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.569540 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzgw\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\") pod \"maas-default-gateway-openshift-default-59f957fdd7-8xwk6\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:17.686603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:17.686549 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:18.109504 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:28:18.109466 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5c4d0d_59b9_4e15_8d50_f14e853846a8.slice/crio-b2e8c521903d03b0cf8bfbac2770004da81bbca109b55a56d52d37b99869d8ff WatchSource:0}: Error finding container b2e8c521903d03b0cf8bfbac2770004da81bbca109b55a56d52d37b99869d8ff: Status 404 returned error can't find the container with id b2e8c521903d03b0cf8bfbac2770004da81bbca109b55a56d52d37b99869d8ff Nov 03 01:28:18.111596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.111547 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:18.111699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.111641 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:18.111759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.111708 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:18.111759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.111735 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:28:18.651918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.651881 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" event={"ID":"6a5c4d0d-59b9-4e15-8d50-f14e853846a8","Type":"ContainerStarted","Data":"a71d3e80649ac339c0ddf2fcfe0ff7df99d65aa0c6c7c4e08cf08c814dd62f7a"} Nov 03 01:28:18.652301 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.651935 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" event={"ID":"6a5c4d0d-59b9-4e15-8d50-f14e853846a8","Type":"ContainerStarted","Data":"b2e8c521903d03b0cf8bfbac2770004da81bbca109b55a56d52d37b99869d8ff"} Nov 03 01:28:18.690575 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.689640 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:18.781582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:18.780638 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podStartSLOduration=1.780620039 podStartE2EDuration="1.780620039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:18.779988341 +0000 UTC m=+634.358810928" watchObservedRunningTime="2026-11-03 01:28:18.780620039 +0000 UTC m=+634.359442626" Nov 03 01:28:19.693707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:19.693670 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:19.694298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:19.694266 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:20.687042 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:20.687002 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:20.687355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:20.687326 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:21.687044 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:21.687000 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:21.687467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:21.687063 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:22.687702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:22.687661 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:22.688116 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:22.687730 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:23.686988 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:23.686956 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:23.687361 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:23.687328 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:24.687845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:24.687809 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 01:28:24.688226 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:24.687869 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 01:28:26.688202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:26.688159 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:28:26.688625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:26.688224 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:28:27.729811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:27.729773 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:28:27.730302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:27.729831 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:28:28.226297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:28.226254 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:28.226946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:28.226922 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:28.227501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:28.227484 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:28:32.149462 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:28:32.149428 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:28:32.488483 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.488451 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 01:28:32.492001 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.491983 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.563866 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.563833 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.563866 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.563869 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.563888 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564029 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564059 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564169 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564129 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc7j\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564209 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564190 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564228 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.564277 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.564249 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.601275 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.601241 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 01:28:32.664733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664697 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664742 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664761 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664820 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664847 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664872 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.664934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664910 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664937 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.664996 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc7j\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.665204 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.665289 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.665364 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665654 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.665632 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.665849 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.665831 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.667258 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.667239 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.667477 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.667459 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.728169 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.728130 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.728341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.728220 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc7j\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-74bnn\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:32.805127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:32.805047 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:33.231639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.231612 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 01:28:33.232813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.232782 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:33.232872 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.232845 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:33.232906 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.232872 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:28:33.773580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.772600 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" event={"ID":"9db16e50-75ce-46cb-ac39-01e9b20e4625","Type":"ContainerStarted","Data":"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af"} Nov 03 01:28:33.773580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.772703 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" event={"ID":"9db16e50-75ce-46cb-ac39-01e9b20e4625","Type":"ContainerStarted","Data":"c1852fd4e88112562763d5a5e27fc1b31298e747cdb2feb98ced8f99e7a80113"} Nov 03 01:28:33.805496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.805460 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:33.898289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:33.898230 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podStartSLOduration=1.8982165279999998 podStartE2EDuration="1.898216528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:33.896924202 +0000 UTC m=+649.475746771" watchObservedRunningTime="2026-11-03 01:28:33.898216528 +0000 UTC m=+649.477039115" Nov 03 01:28:34.807040 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:34.807001 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:34.807448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:34.807055 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:35.806265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:35.806229 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:35.806440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:35.806281 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:36.806253 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:36.806218 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:36.806667 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:36.806270 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:37.806300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:37.806262 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:37.806715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:37.806315 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:38.805389 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:38.805360 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:38.805579 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:38.805414 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:39.805677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:39.805638 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Nov 03 01:28:39.806199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:39.805701 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Nov 03 01:28:41.481059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:41.481027 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:41.481804 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:41.481770 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:41.482569 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:41.482542 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 01:28:43.665626 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.665592 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:28:43.666079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.665774 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57c96bc49d-gghrk" podUID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" containerName="authorino" containerID="cri-o://b3c8074dad3936e7fc6dfc1f2ba4503087f4c3343de197f91cab6cf3780458ed" gracePeriod=30 Nov 03 01:28:43.836161 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.836129 4992 generic.go:356] "Generic (PLEG): container finished" podID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" containerID="b3c8074dad3936e7fc6dfc1f2ba4503087f4c3343de197f91cab6cf3780458ed" exitCode=0 Nov 03 01:28:43.836302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.836188 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c96bc49d-gghrk" event={"ID":"d497c97c-628a-4bb7-a663-bc97d1aae1d2","Type":"ContainerDied","Data":"b3c8074dad3936e7fc6dfc1f2ba4503087f4c3343de197f91cab6cf3780458ed"} Nov 03 01:28:43.908978 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.908948 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:28:43.979344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.979263 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\" (UniqueName: \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\") pod \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " Nov 03 01:28:43.979344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.979321 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\" (UniqueName: \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\") pod \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\" (UID: \"d497c97c-628a-4bb7-a663-bc97d1aae1d2\") " Nov 03 01:28:43.981437 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.981402 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz" pod "d497c97c-628a-4bb7-a663-bc97d1aae1d2" (UID: "d497c97c-628a-4bb7-a663-bc97d1aae1d2"). InnerVolumeSpecName "kube-api-access-24dsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:43.991055 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:43.991028 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert" pod "d497c97c-628a-4bb7-a663-bc97d1aae1d2" (UID: "d497c97c-628a-4bb7-a663-bc97d1aae1d2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:28:44.080682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:44.080642 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dsz\" (UniqueName: \"kubernetes.io/projected/d497c97c-628a-4bb7-a663-bc97d1aae1d2-kube-api-access-24dsz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.080682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:44.080674 4992 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d497c97c-628a-4bb7-a663-bc97d1aae1d2-tls-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:28:44.841661 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:44.841632 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c96bc49d-gghrk" Nov 03 01:28:44.842110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:44.841668 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c96bc49d-gghrk" event={"ID":"d497c97c-628a-4bb7-a663-bc97d1aae1d2","Type":"ContainerDied","Data":"41ef4dc83640b59fa50852e17f96ef6c4926d41dd031059dfd41506f957ba087"} Nov 03 01:28:44.842110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:44.841721 4992 scope.go:122] "RemoveContainer" containerID="b3c8074dad3936e7fc6dfc1f2ba4503087f4c3343de197f91cab6cf3780458ed" Nov 03 01:28:45.013104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:45.013066 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:28:45.064716 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:45.064681 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57c96bc49d-gghrk"] Nov 03 01:28:46.593355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:28:46.593322 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" path="/var/lib/kubelet/pods/d497c97c-628a-4bb7-a663-bc97d1aae1d2/volumes" Nov 03 01:29:02.150264 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:02.150186 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:29:29.192282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.192243 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:29:29.192781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.192672 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" containerName="authorino" Nov 03 01:29:29.192781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.192684 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" containerName="authorino" Nov 03 01:29:29.192859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.192792 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d497c97c-628a-4bb7-a663-bc97d1aae1d2" containerName="authorino" Nov 03 01:29:29.195828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.195812 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.251990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.251958 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 01:29:29.279581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.279526 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.279742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.279670 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.279742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.279715 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcvt\" (UniqueName: \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.279887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.279863 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.303848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.303803 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-98c7cd16-serving-cert\"" Nov 03 01:29:29.304021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.303862 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 01:29:29.304021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.303905 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\"" Nov 03 01:29:29.304021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.303952 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 01:29:29.306396 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.306367 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:29:29.381351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.381323 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcvt\" (UniqueName: \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.381518 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.381377 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.381518 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.381431 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.381518 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.381486 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.381710 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:29.381535 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-serving-cert: secret "maas-api-e2e-ait-98c7cd16-serving-cert" not found Nov 03 01:29:29.381710 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:29.381633 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls podName:3dfdf3b6-8f84-4f68-9254-94e8dd54dacc nodeName:}" failed. No retries permitted until 2026-11-03 01:29:29.881615124 +0000 UTC m=+705.460437702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls") pod "maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" (UID: "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc") : secret "maas-api-e2e-ait-98c7cd16-serving-cert" not found Nov 03 01:29:29.382273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.382250 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.383966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.383945 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.445766 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.445700 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcvt\" (UniqueName: \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.887509 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.887426 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:29.889918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:29.889895 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") pod \"maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:30.108692 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:30.108660 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:30.483252 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:29:30.483221 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfdf3b6_8f84_4f68_9254_94e8dd54dacc.slice/crio-10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07 WatchSource:0}: Error finding container 10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07: Status 404 returned error can't find the container with id 10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07 Nov 03 01:29:30.491345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:30.485954 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:29:30.492452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:30.492421 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:29:31.069960 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:31.069917 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" event={"ID":"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc","Type":"ContainerStarted","Data":"10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07"} Nov 03 01:29:32.077545 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:32.077501 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" event={"ID":"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc","Type":"ContainerStarted","Data":"2350ece34c6c5cc371c6267ccb0b008b1c755338bd03d8894ead2edc77d74ab6"} Nov 03 01:29:32.077979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:32.077690 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:32.150780 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:32.150746 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:29:32.203029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:32.202972 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" podStartSLOduration=2.239518753 podStartE2EDuration="3.202958012s" totalImagesPullingTime="963.439259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:32.202324812 +0000 UTC m=+707.781147400" watchObservedRunningTime="2026-11-03 01:29:32.202958012 +0000 UTC m=+707.781780599" Nov 03 01:29:34.380668 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.380633 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:29:34.384439 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.384416 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.494377 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.494339 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:29:34.503896 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.503866 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e3f0a557-serving-cert\"" Nov 03 01:29:34.531512 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.531475 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.531676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.531520 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.531676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.531619 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.531676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.531642 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddfd\" (UniqueName: \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.632680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.632534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.632680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.632651 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddfd\" (UniqueName: \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.632996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.632717 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.632996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.632772 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.632996 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:34.632823 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-serving-cert: secret "maas-api-e2e-migrate-e3f0a557-serving-cert" not found Nov 03 01:29:34.632996 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:34.632906 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls podName:b75cf0b7-b5f1-432b-802f-2661222a78a0 nodeName:}" failed. No retries permitted until 2026-11-03 01:29:35.132885049 +0000 UTC m=+710.711707628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls") pod "maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" (UID: "b75cf0b7-b5f1-432b-802f-2661222a78a0") : secret "maas-api-e2e-migrate-e3f0a557-serving-cert" not found Nov 03 01:29:34.633345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.633326 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.635363 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.635343 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:34.712519 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:34.712490 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddfd\" (UniqueName: \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:35.138327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:35.138283 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:35.140747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:35.140723 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") pod \"maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:35.303624 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:35.303581 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:35.673724 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:29:35.673671 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75cf0b7_b5f1_432b_802f_2661222a78a0.slice/crio-b6b7e6362d4c13608562b3bd5bf4644c0159f80002f77477e19b96a67f49b0d6 WatchSource:0}: Error finding container b6b7e6362d4c13608562b3bd5bf4644c0159f80002f77477e19b96a67f49b0d6: Status 404 returned error can't find the container with id b6b7e6362d4c13608562b3bd5bf4644c0159f80002f77477e19b96a67f49b0d6 Nov 03 01:29:35.675982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:35.675953 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:29:36.099662 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:36.099583 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" event={"ID":"b75cf0b7-b5f1-432b-802f-2661222a78a0","Type":"ContainerStarted","Data":"b6b7e6362d4c13608562b3bd5bf4644c0159f80002f77477e19b96a67f49b0d6"} Nov 03 01:29:37.106699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.106664 4992 generic.go:356] "Generic (PLEG): container finished" podID="f592be42-f99a-48e7-adb0-314512784195" containerID="d7a6c1215a9b6407218b508a94b18a12b053bf8f3b45ec5fc1e2481c9a3aba32" exitCode=1 Nov 03 01:29:37.107166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.106747 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerDied","Data":"d7a6c1215a9b6407218b508a94b18a12b053bf8f3b45ec5fc1e2481c9a3aba32"} Nov 03 01:29:37.107166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.107160 4992 scope.go:122] "RemoveContainer" containerID="d7a6c1215a9b6407218b508a94b18a12b053bf8f3b45ec5fc1e2481c9a3aba32" Nov 03 01:29:37.108472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.108453 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" event={"ID":"b75cf0b7-b5f1-432b-802f-2661222a78a0","Type":"ContainerStarted","Data":"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43"} Nov 03 01:29:37.108532 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.108508 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:37.347936 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.347886 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" podStartSLOduration=2.525836833 podStartE2EDuration="3.347873305s" totalImagesPullingTime="822.036472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:37.347481854 +0000 UTC m=+712.926304439" watchObservedRunningTime="2026-11-03 01:29:37.347873305 +0000 UTC m=+712.926695890" Nov 03 01:29:37.741112 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:37.741079 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:29:38.089309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:38.089281 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:29:38.115405 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:38.115369 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerStarted","Data":"2643013ebd7f0160dafe7b7b8e8747cbc078d38172d3f92657620ae647be53c5"} Nov 03 01:29:38.115805 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:38.115518 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:29:39.192130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.192093 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:29:39.195782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.195763 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.250107 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.250075 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fc87c747-serving-cert\"" Nov 03 01:29:39.278629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.278592 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.278629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.278633 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.278848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.278782 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkm6\" (UniqueName: \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.278897 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.278867 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.303990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.303957 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:29:39.380092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.380052 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkm6\" (UniqueName: \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.380253 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.380210 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.380338 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.380281 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.380338 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.380311 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.380478 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:39.380462 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-serving-cert: secret "maas-api-e2e-ait-fc87c747-serving-cert" not found Nov 03 01:29:39.380540 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:39.380530 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls podName:50611e46-6f57-4008-8069-a049f243d14e nodeName:}" failed. No retries permitted until 2026-11-03 01:29:39.880509619 +0000 UTC m=+715.459332197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls") pod "maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" (UID: "50611e46-6f57-4008-8069-a049f243d14e") : secret "maas-api-e2e-ait-fc87c747-serving-cert" not found Nov 03 01:29:39.380877 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.380854 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.382768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.382744 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.443376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.443307 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkm6\" (UniqueName: \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.886520 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.886423 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:39.888813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:39.888785 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") pod \"maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:40.109354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:40.109318 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:40.482091 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:29:40.482059 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50611e46_6f57_4008_8069_a049f243d14e.slice/crio-762a0012328e09abd57ecaabded275dc70a2d7aa6bbeb60b14d6e542f7523039 WatchSource:0}: Error finding container 762a0012328e09abd57ecaabded275dc70a2d7aa6bbeb60b14d6e542f7523039: Status 404 returned error can't find the container with id 762a0012328e09abd57ecaabded275dc70a2d7aa6bbeb60b14d6e542f7523039 Nov 03 01:29:40.484107 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:40.484080 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:29:41.139972 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:41.139940 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerStarted","Data":"762a0012328e09abd57ecaabded275dc70a2d7aa6bbeb60b14d6e542f7523039"} Nov 03 01:29:42.146398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:42.146366 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerStarted","Data":"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f"} Nov 03 01:29:42.146839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:42.146440 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:42.268641 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:42.268384 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" podStartSLOduration=2.360316053 podStartE2EDuration="3.268365831s" totalImagesPullingTime="908.049778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:42.266955631 +0000 UTC m=+717.845778218" watchObservedRunningTime="2026-11-03 01:29:42.268365831 +0000 UTC m=+717.847188419" Nov 03 01:29:43.120817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:43.120780 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:29:48.153168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:48.153112 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.55:8443/health\": dial tcp 10.132.0.55:8443: connect: connection refused" Nov 03 01:29:52.196390 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:52.196355 4992 generic.go:356] "Generic (PLEG): container finished" podID="50611e46-6f57-4008-8069-a049f243d14e" containerID="910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f" exitCode=1 Nov 03 01:29:52.196791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:52.196406 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerDied","Data":"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f"} Nov 03 01:29:52.196791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:52.196773 4992 scope.go:122] "RemoveContainer" containerID="910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f" Nov 03 01:29:53.203020 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:53.202983 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerStarted","Data":"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821"} Nov 03 01:29:53.203370 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:53.203209 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:29:54.003199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.003164 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:29:54.006862 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.006846 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.012355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.012336 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.012459 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.012394 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.012508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.012487 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzgpr\" (UniqueName: \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.012663 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.012644 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.061677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.061644 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:29:54.110965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.110936 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1e406323-serving-cert\"" Nov 03 01:29:54.113278 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.113255 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.113393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.113302 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.113393 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:54.113386 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-serving-cert: secret "maas-api-e2e-derive-1e406323-serving-cert" not found Nov 03 01:29:54.113497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.113406 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzgpr\" (UniqueName: \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.113497 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:29:54.113444 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls podName:eb40bb53-1e1d-4255-bfaf-b626e1ce22fc nodeName:}" failed. No retries permitted until 2026-11-03 01:29:54.613423836 +0000 UTC m=+730.192246420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls") pod "maas-api-e2e-derive-1e406323-7969bf9644-m56vj" (UID: "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc") : secret "maas-api-e2e-derive-1e406323-serving-cert" not found Nov 03 01:29:54.113629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.113547 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.113992 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.113970 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.115726 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.115706 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.176395 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.176362 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzgpr\" (UniqueName: \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.618905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.618865 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.621410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.621382 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") pod \"maas-api-e2e-derive-1e406323-7969bf9644-m56vj\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:54.919916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:54.919883 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:55.292761 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:29:55.292724 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb40bb53_1e1d_4255_bfaf_b626e1ce22fc.slice/crio-5a834a823ffd9c61176432438f41bf0cd2fac266b74a0fc3e8b321cd6a0d1c47 WatchSource:0}: Error finding container 5a834a823ffd9c61176432438f41bf0cd2fac266b74a0fc3e8b321cd6a0d1c47: Status 404 returned error can't find the container with id 5a834a823ffd9c61176432438f41bf0cd2fac266b74a0fc3e8b321cd6a0d1c47 Nov 03 01:29:55.294305 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:55.294277 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:29:56.233270 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:56.233231 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" event={"ID":"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc","Type":"ContainerStarted","Data":"5a834a823ffd9c61176432438f41bf0cd2fac266b74a0fc3e8b321cd6a0d1c47"} Nov 03 01:29:57.239265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:57.239224 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" event={"ID":"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc","Type":"ContainerStarted","Data":"f3eee18e69272da6814d8509402cb7bfbd2a76eeb648df270a48e36328472d05"} Nov 03 01:29:57.239694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:57.239286 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:29:57.360728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:57.360674 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" podStartSLOduration=3.443467303 podStartE2EDuration="4.360660246s" totalImagesPullingTime="917.192943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:57.359770996 +0000 UTC m=+732.938593584" watchObservedRunningTime="2026-11-03 01:29:57.360660246 +0000 UTC m=+732.939482829" Nov 03 01:29:59.214733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:29:59.214701 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:30:02.149784 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:02.149748 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:30:03.253322 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:03.253291 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:30:09.132233 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:09.132202 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:30:29.151965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.151930 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:30:29.152374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.152121 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" podUID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" containerName="maas-api" containerID="cri-o://2350ece34c6c5cc371c6267ccb0b008b1c755338bd03d8894ead2edc77d74ab6" gracePeriod=30 Nov 03 01:30:29.402964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.402792 4992 generic.go:356] "Generic (PLEG): container finished" podID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" containerID="2350ece34c6c5cc371c6267ccb0b008b1c755338bd03d8894ead2edc77d74ab6" exitCode=0 Nov 03 01:30:29.402964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.402859 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" event={"ID":"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc","Type":"ContainerDied","Data":"2350ece34c6c5cc371c6267ccb0b008b1c755338bd03d8894ead2edc77d74ab6"} Nov 03 01:30:29.402964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.402912 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" event={"ID":"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc","Type":"ContainerDied","Data":"10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07"} Nov 03 01:30:29.402964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.402929 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ad3ce6542e2cfbee6498a4c5b45c344cdbbf98af3ec121a01086bbd71dfe07" Nov 03 01:30:29.420140 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.420114 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:30:29.557714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.557684 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") pod \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " Nov 03 01:30:29.557913 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.557762 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\") pod \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " Nov 03 01:30:29.557913 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.557783 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\" (UniqueName: \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\") pod \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " Nov 03 01:30:29.557913 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.557811 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\") pod \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\" (UID: \"3dfdf3b6-8f84-4f68-9254-94e8dd54dacc\") " Nov 03 01:30:29.558288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.558257 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca" pod "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" (UID: "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:29.559909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.559881 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls" pod "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" (UID: "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:29.560008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.559951 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs" pod "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" (UID: "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:29.560008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.559986 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt" pod "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" (UID: "3dfdf3b6-8f84-4f68-9254-94e8dd54dacc"). InnerVolumeSpecName "kube-api-access-qpcvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:29.658952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.658870 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:29.658952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.658905 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:29.658952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.658916 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpcvt\" (UniqueName: \"kubernetes.io/projected/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-kube-api-access-qpcvt\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:29.658952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:29.658926 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:30.296142 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.296105 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:30:30.296617 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.296295 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" podUID="b75cf0b7-b5f1-432b-802f-2661222a78a0" containerName="maas-api" containerID="cri-o://50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43" gracePeriod=30 Nov 03 01:30:30.407406 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.407377 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b" Nov 03 01:30:30.557683 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.557625 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:30:30.572062 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.572021 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:30:30.577818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.577782 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-98c7cd16-55cf8448c6-wns2b"] Nov 03 01:30:30.594200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.594168 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" path="/var/lib/kubelet/pods/3dfdf3b6-8f84-4f68-9254-94e8dd54dacc/volumes" Nov 03 01:30:30.668652 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.668621 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\" (UniqueName: \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\") pod \"b75cf0b7-b5f1-432b-802f-2661222a78a0\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " Nov 03 01:30:30.668828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.668693 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") pod \"b75cf0b7-b5f1-432b-802f-2661222a78a0\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " Nov 03 01:30:30.668828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.668771 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\") pod \"b75cf0b7-b5f1-432b-802f-2661222a78a0\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " Nov 03 01:30:30.668828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.668795 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\") pod \"b75cf0b7-b5f1-432b-802f-2661222a78a0\" (UID: \"b75cf0b7-b5f1-432b-802f-2661222a78a0\") " Nov 03 01:30:30.669165 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.669132 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca" pod "b75cf0b7-b5f1-432b-802f-2661222a78a0" (UID: "b75cf0b7-b5f1-432b-802f-2661222a78a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:30.671042 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.671008 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls" pod "b75cf0b7-b5f1-432b-802f-2661222a78a0" (UID: "b75cf0b7-b5f1-432b-802f-2661222a78a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:30.671143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.671019 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs" pod "b75cf0b7-b5f1-432b-802f-2661222a78a0" (UID: "b75cf0b7-b5f1-432b-802f-2661222a78a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:30.671143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.671072 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd" pod "b75cf0b7-b5f1-432b-802f-2661222a78a0" (UID: "b75cf0b7-b5f1-432b-802f-2661222a78a0"). InnerVolumeSpecName "kube-api-access-kddfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:30.769891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.769832 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75cf0b7-b5f1-432b-802f-2661222a78a0-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:30.769891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.769887 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:30.770100 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.769902 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kddfd\" (UniqueName: \"kubernetes.io/projected/b75cf0b7-b5f1-432b-802f-2661222a78a0-kube-api-access-kddfd\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:30.770100 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:30.769915 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75cf0b7-b5f1-432b-802f-2661222a78a0-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.413711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.413675 4992 generic.go:356] "Generic (PLEG): container finished" podID="b75cf0b7-b5f1-432b-802f-2661222a78a0" containerID="50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43" exitCode=0 Nov 03 01:30:31.414168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.413720 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" Nov 03 01:30:31.414168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.413730 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" event={"ID":"b75cf0b7-b5f1-432b-802f-2661222a78a0","Type":"ContainerDied","Data":"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43"} Nov 03 01:30:31.414168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.413778 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz" event={"ID":"b75cf0b7-b5f1-432b-802f-2661222a78a0","Type":"ContainerDied","Data":"b6b7e6362d4c13608562b3bd5bf4644c0159f80002f77477e19b96a67f49b0d6"} Nov 03 01:30:31.414168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.413799 4992 scope.go:122] "RemoveContainer" containerID="50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43" Nov 03 01:30:31.423441 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.423425 4992 scope.go:122] "RemoveContainer" containerID="50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43" Nov 03 01:30:31.423715 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:31.423696 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43\": container with ID starting with 50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43 not found: ID does not exist" containerID="50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43" Nov 03 01:30:31.423786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.423726 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43"} err="failed to get container status \"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43\": rpc error: code = NotFound desc = could not find container \"50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43\": container with ID starting with 50780b13dfba7d86ec5011e9d76de7e5175a065d4e5582626101ac9e271f2a43 not found: ID does not exist" Nov 03 01:30:31.522128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.522096 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:30:31.522328 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.522307 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" containerID="cri-o://42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821" gracePeriod=30 Nov 03 01:30:31.587746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.587713 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:30:31.592439 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.592407 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3f0a557-79ff8c568-6jzrz"] Nov 03 01:30:31.750274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750241 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l"] Nov 03 01:30:31.750681 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750667 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b75cf0b7-b5f1-432b-802f-2661222a78a0" containerName="maas-api" Nov 03 01:30:31.750759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750682 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75cf0b7-b5f1-432b-802f-2661222a78a0" containerName="maas-api" Nov 03 01:30:31.750759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750705 4992 cpu_manager.go:414] "containerMap: removing container" podUID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" containerName="maas-api" Nov 03 01:30:31.750759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750710 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" containerName="maas-api" Nov 03 01:30:31.750918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750788 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75cf0b7-b5f1-432b-802f-2661222a78a0" containerName="maas-api" Nov 03 01:30:31.750918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.750799 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfdf3b6-8f84-4f68-9254-94e8dd54dacc" containerName="maas-api" Nov 03 01:30:31.753975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.753953 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.787022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.786992 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:30:31.808656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.808626 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l"] Nov 03 01:30:31.882369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882340 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\" (UniqueName: \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\") pod \"50611e46-6f57-4008-8069-a049f243d14e\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " Nov 03 01:30:31.882516 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882383 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\") pod \"50611e46-6f57-4008-8069-a049f243d14e\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " Nov 03 01:30:31.882516 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882448 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\") pod \"50611e46-6f57-4008-8069-a049f243d14e\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " Nov 03 01:30:31.882516 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882495 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") pod \"50611e46-6f57-4008-8069-a049f243d14e\" (UID: \"50611e46-6f57-4008-8069-a049f243d14e\") " Nov 03 01:30:31.882741 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882720 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.882787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882772 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pvb\" (UniqueName: \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.882926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882895 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca" pod "50611e46-6f57-4008-8069-a049f243d14e" (UID: "50611e46-6f57-4008-8069-a049f243d14e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:31.883058 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.882976 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.883138 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.883106 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50611e46-6f57-4008-8069-a049f243d14e-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.884611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.884582 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs" pod "50611e46-6f57-4008-8069-a049f243d14e" (UID: "50611e46-6f57-4008-8069-a049f243d14e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:31.884611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.884604 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls" pod "50611e46-6f57-4008-8069-a049f243d14e" (UID: "50611e46-6f57-4008-8069-a049f243d14e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:31.884747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.884695 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6" pod "50611e46-6f57-4008-8069-a049f243d14e" (UID: "50611e46-6f57-4008-8069-a049f243d14e"). InnerVolumeSpecName "kube-api-access-xkkm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:31.886360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.886344 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:30:31.886656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.886643 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 01:30:31.984552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984476 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.984552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984517 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pvb\" (UniqueName: \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.984786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984627 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.984786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984672 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.984786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984687 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkm6\" (UniqueName: \"kubernetes.io/projected/50611e46-6f57-4008-8069-a049f243d14e-kube-api-access-xkkm6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.984786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.984702 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50611e46-6f57-4008-8069-a049f243d14e-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:31.985216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.985195 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:31.986744 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:31.986725 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:32.047572 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.047529 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pvb\" (UniqueName: \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\") pod \"maas-api-revoke-keys-e2e-derive-1e406323-k599l\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:32.085358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.085327 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:32.157945 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:32.157911 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:30:32.425460 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.425428 4992 generic.go:356] "Generic (PLEG): container finished" podID="50611e46-6f57-4008-8069-a049f243d14e" containerID="42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821" exitCode=0 Nov 03 01:30:32.425946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.425496 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerDied","Data":"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821"} Nov 03 01:30:32.425946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.425501 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" Nov 03 01:30:32.425946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.425583 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m" event={"ID":"50611e46-6f57-4008-8069-a049f243d14e","Type":"ContainerDied","Data":"762a0012328e09abd57ecaabded275dc70a2d7aa6bbeb60b14d6e542f7523039"} Nov 03 01:30:32.425946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.425605 4992 scope.go:122] "RemoveContainer" containerID="42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821" Nov 03 01:30:32.438912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.438894 4992 scope.go:122] "RemoveContainer" containerID="910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f" Nov 03 01:30:32.447124 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.447091 4992 scope.go:122] "RemoveContainer" containerID="42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821" Nov 03 01:30:32.447439 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:32.447419 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821\": container with ID starting with 42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821 not found: ID does not exist" containerID="42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821" Nov 03 01:30:32.447491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.447448 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821"} err="failed to get container status \"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821\": rpc error: code = NotFound desc = could not find container \"42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821\": container with ID starting with 42e1ffcb982d66b7791da53ed97853f0d18752c4ba750474ded44817a9071821 not found: ID does not exist" Nov 03 01:30:32.447491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.447465 4992 scope.go:122] "RemoveContainer" containerID="910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f" Nov 03 01:30:32.447745 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:32.447730 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f\": container with ID starting with 910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f not found: ID does not exist" containerID="910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f" Nov 03 01:30:32.447796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.447750 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f"} err="failed to get container status \"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f\": rpc error: code = NotFound desc = could not find container \"910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f\": container with ID starting with 910b3d6650da3d0d5dab6b0dfd634342b9688fa3ed78d0179436bb0667b7200f not found: ID does not exist" Nov 03 01:30:32.457594 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:30:32.457528 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccfe377_1b71_48c3_8db5_010e66b56805.slice/crio-4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8 WatchSource:0}: Error finding container 4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8: Status 404 returned error can't find the container with id 4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8 Nov 03 01:30:32.459598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.459550 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l"] Nov 03 01:30:32.596305 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.596263 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75cf0b7-b5f1-432b-802f-2661222a78a0" path="/var/lib/kubelet/pods/b75cf0b7-b5f1-432b-802f-2661222a78a0/volumes" Nov 03 01:30:32.596602 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.596587 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:30:32.607974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:32.607942 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc87c747-7c77c4546b-lzl6m"] Nov 03 01:30:33.431820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:33.431781 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" event={"ID":"0ccfe377-1b71-48c3-8db5-010e66b56805","Type":"ContainerStarted","Data":"4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8"} Nov 03 01:30:34.184239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184196 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 01:30:34.184724 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184704 4992 cpu_manager.go:414] "containerMap: removing container" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.184822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184724 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.184822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184745 4992 cpu_manager.go:414] "containerMap: removing container" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.184822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184753 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.184972 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184899 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.184972 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.184912 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="50611e46-6f57-4008-8069-a049f243d14e" containerName="maas-api" Nov 03 01:30:34.189380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.189359 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.246430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.246398 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 01:30:34.294913 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.294884 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\"" Nov 03 01:30:34.310601 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310407 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310601 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310496 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310601 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310546 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6hx\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310601 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310592 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310615 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310660 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310682 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310739 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.310939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.310788 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411539 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411494 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411706 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411598 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411706 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411656 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411702 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6hx\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411730 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411756 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.411811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411800 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411826 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.411884 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412123 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.412095 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.412273 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.412345 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.412610 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.412765 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.412661 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.414020 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.413991 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.414382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.414362 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.477029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.476954 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6hx\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.477029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.476954 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.502505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.502473 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:34.600500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.600458 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50611e46-6f57-4008-8069-a049f243d14e" path="/var/lib/kubelet/pods/50611e46-6f57-4008-8069-a049f243d14e/volumes" Nov 03 01:30:34.938306 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:30:34.938278 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b27a046_e80f_43d0_83de_5cac014ef0bc.slice/crio-e8aa329b9b985705cf6b133ea1a951b672a6d6e7bd894713fe9e7ecfddd81002 WatchSource:0}: Error finding container e8aa329b9b985705cf6b133ea1a951b672a6d6e7bd894713fe9e7ecfddd81002: Status 404 returned error can't find the container with id e8aa329b9b985705cf6b133ea1a951b672a6d6e7bd894713fe9e7ecfddd81002 Nov 03 01:30:34.940545 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.940509 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:34.940656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.940594 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:34.940656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.940626 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:30:34.940811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:34.940792 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 01:30:35.444615 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:35.444583 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerStarted","Data":"1194fb66238bc601fc5d722db79fc31bafd587ac5e45983d8c9ae4991f048553"} Nov 03 01:30:35.444800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:35.444637 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerStarted","Data":"e8aa329b9b985705cf6b133ea1a951b672a6d6e7bd894713fe9e7ecfddd81002"} Nov 03 01:30:35.502928 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:35.502896 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:35.510551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:35.510521 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:35.579223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:35.579172 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podStartSLOduration=1.579160479 podStartE2EDuration="1.579160479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:35.577745562 +0000 UTC m=+771.156568148" watchObservedRunningTime="2026-11-03 01:30:35.579160479 +0000 UTC m=+771.157983065" Nov 03 01:30:36.449917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:36.449875 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:36.450843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:36.450825 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:30:38.461799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:38.461767 4992 generic.go:356] "Generic (PLEG): container finished" podID="0ccfe377-1b71-48c3-8db5-010e66b56805" containerID="1e964726c3f06af51cf8f8ce74364b54f1d987f9f4f288878e2118bf14d07c6b" exitCode=0 Nov 03 01:30:38.462222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:38.461833 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" event={"ID":"0ccfe377-1b71-48c3-8db5-010e66b56805","Type":"ContainerDied","Data":"1e964726c3f06af51cf8f8ce74364b54f1d987f9f4f288878e2118bf14d07c6b"} Nov 03 01:30:39.621577 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.621542 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:39.661488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.661451 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\") pod \"0ccfe377-1b71-48c3-8db5-010e66b56805\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " Nov 03 01:30:39.661807 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.661784 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca" pod "0ccfe377-1b71-48c3-8db5-010e66b56805" (UID: "0ccfe377-1b71-48c3-8db5-010e66b56805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:39.762006 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.761906 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\") pod \"0ccfe377-1b71-48c3-8db5-010e66b56805\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " Nov 03 01:30:39.762181 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.762038 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\" (UniqueName: \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\") pod \"0ccfe377-1b71-48c3-8db5-010e66b56805\" (UID: \"0ccfe377-1b71-48c3-8db5-010e66b56805\") " Nov 03 01:30:39.762349 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.762328 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:39.764078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.764048 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca" pod "0ccfe377-1b71-48c3-8db5-010e66b56805" (UID: "0ccfe377-1b71-48c3-8db5-010e66b56805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:30:39.764664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.764620 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb" pod "0ccfe377-1b71-48c3-8db5-010e66b56805" (UID: "0ccfe377-1b71-48c3-8db5-010e66b56805"). InnerVolumeSpecName "kube-api-access-p2pvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:39.862942 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.862904 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ccfe377-1b71-48c3-8db5-010e66b56805-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:39.862942 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:39.862937 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pvb\" (UniqueName: \"kubernetes.io/projected/0ccfe377-1b71-48c3-8db5-010e66b56805-kube-api-access-p2pvb\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:30:40.474247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:40.474214 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" Nov 03 01:30:40.474247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:40.474239 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l" event={"ID":"0ccfe377-1b71-48c3-8db5-010e66b56805","Type":"ContainerDied","Data":"4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8"} Nov 03 01:30:40.474467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:40.474284 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4971397c88ca81d1a5a15979570a6ef65749dd40d2abf718959c706f195de8" Nov 03 01:30:53.885244 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.885202 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 01:30:53.885672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.885658 4992 cpu_manager.go:414] "containerMap: removing container" podUID="0ccfe377-1b71-48c3-8db5-010e66b56805" containerName="revoke-keys" Nov 03 01:30:53.885727 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.885672 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccfe377-1b71-48c3-8db5-010e66b56805" containerName="revoke-keys" Nov 03 01:30:53.885788 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.885780 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccfe377-1b71-48c3-8db5-010e66b56805" containerName="revoke-keys" Nov 03 01:30:53.890390 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.890371 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:53.944075 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.944041 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 01:30:53.993411 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:53.993384 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-41aa5f-serving-cert\"" Nov 03 01:30:54.017689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.017652 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.017689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.017707 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25tm\" (UniqueName: \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.017945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.017856 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.017945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.017907 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.119513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.119479 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.119732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.119555 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.119732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.119669 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.119732 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:54.119683 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-serving-cert: secret "maas-api-e2e-worker-w3-41aa5f-serving-cert" not found Nov 03 01:30:54.119732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.119698 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25tm\" (UniqueName: \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.119939 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:30:54.119780 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls podName:9703362e-f746-4a90-8dba-e1fe1368d960 nodeName:}" failed. No retries permitted until 2026-11-03 01:30:54.619758846 +0000 UTC m=+790.198581427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls") pod "maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" (UID: "9703362e-f746-4a90-8dba-e1fe1368d960") : secret "maas-api-e2e-worker-w3-41aa5f-serving-cert" not found Nov 03 01:30:54.120816 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.120795 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.130454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.130424 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.183106 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.183076 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25tm\" (UniqueName: \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.624844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.624753 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.627263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.627240 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") pod \"maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:54.803652 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:54.803612 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:55.177102 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:30:55.176349 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9703362e_f746_4a90_8dba_e1fe1368d960.slice/crio-9092e23e9929d1f079cbfe01307fa902db13d98c174fe0e57ef6511083e58edd WatchSource:0}: Error finding container 9092e23e9929d1f079cbfe01307fa902db13d98c174fe0e57ef6511083e58edd: Status 404 returned error can't find the container with id 9092e23e9929d1f079cbfe01307fa902db13d98c174fe0e57ef6511083e58edd Nov 03 01:30:55.178650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:55.178628 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 01:30:55.556440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:55.556359 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" event={"ID":"9703362e-f746-4a90-8dba-e1fe1368d960","Type":"ContainerStarted","Data":"9092e23e9929d1f079cbfe01307fa902db13d98c174fe0e57ef6511083e58edd"} Nov 03 01:30:56.563063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:56.563023 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" event={"ID":"9703362e-f746-4a90-8dba-e1fe1368d960","Type":"ContainerStarted","Data":"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566"} Nov 03 01:30:56.563458 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:56.563188 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:30:56.683761 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:30:56.683706 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" podStartSLOduration=2.721163971 podStartE2EDuration="3.683691334s" totalImagesPullingTime="962.527363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:56.683300497 +0000 UTC m=+792.262123089" watchObservedRunningTime="2026-11-03 01:30:56.683691334 +0000 UTC m=+792.262513943" Nov 03 01:31:02.149932 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:31:02.149896 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:31:02.574871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:02.574839 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 01:31:10.513345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.513310 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:31:10.513968 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.513526 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" podUID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" containerName="maas-api" containerID="cri-o://f3eee18e69272da6814d8509402cb7bfbd2a76eeb648df270a48e36328472d05" gracePeriod=30 Nov 03 01:31:10.642385 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.642354 4992 generic.go:356] "Generic (PLEG): container finished" podID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" containerID="f3eee18e69272da6814d8509402cb7bfbd2a76eeb648df270a48e36328472d05" exitCode=0 Nov 03 01:31:10.642527 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.642433 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" event={"ID":"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc","Type":"ContainerDied","Data":"f3eee18e69272da6814d8509402cb7bfbd2a76eeb648df270a48e36328472d05"} Nov 03 01:31:10.792316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.792291 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:31:10.884650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.884613 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\") pod \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " Nov 03 01:31:10.884843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.884725 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") pod \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " Nov 03 01:31:10.884843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.884766 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\") pod \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " Nov 03 01:31:10.884843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.884817 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\" (UniqueName: \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\") pod \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\" (UID: \"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc\") " Nov 03 01:31:10.886860 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.886813 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca" pod "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" (UID: "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:10.888183 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.888152 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls" pod "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" (UID: "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:10.888370 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.888349 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs" pod "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" (UID: "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:10.888967 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.888940 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr" pod "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" (UID: "eb40bb53-1e1d-4255-bfaf-b626e1ce22fc"). InnerVolumeSpecName "kube-api-access-tzgpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:10.985930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.985896 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:10.985930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.985925 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:10.985930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.985936 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:10.986195 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:10.985945 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzgpr\" (UniqueName: \"kubernetes.io/projected/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc-kube-api-access-tzgpr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:31:11.649503 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:11.649468 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" Nov 03 01:31:11.649994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:11.649474 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj" event={"ID":"eb40bb53-1e1d-4255-bfaf-b626e1ce22fc","Type":"ContainerDied","Data":"5a834a823ffd9c61176432438f41bf0cd2fac266b74a0fc3e8b321cd6a0d1c47"} Nov 03 01:31:11.649994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:11.649617 4992 scope.go:122] "RemoveContainer" containerID="f3eee18e69272da6814d8509402cb7bfbd2a76eeb648df270a48e36328472d05" Nov 03 01:31:11.821447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:11.821409 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:31:11.873767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:11.873712 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1e406323-7969bf9644-m56vj"] Nov 03 01:31:12.593926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:12.593885 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" path="/var/lib/kubelet/pods/eb40bb53-1e1d-4255-bfaf-b626e1ce22fc/volumes" Nov 03 01:31:12.698288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:12.698253 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l"] Nov 03 01:31:12.761470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:12.751315 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1e406323-k599l"] Nov 03 01:31:14.227364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.227333 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:31:14.227852 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.227836 4992 cpu_manager.go:414] "containerMap: removing container" podUID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" containerName="maas-api" Nov 03 01:31:14.227893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.227853 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" containerName="maas-api" Nov 03 01:31:14.227935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.227925 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb40bb53-1e1d-4255-bfaf-b626e1ce22fc" containerName="maas-api" Nov 03 01:31:14.233039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.233019 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.319754 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.309069 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.320852 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.320915 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.320951 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r848b\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.320985 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.321010 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.321033 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.321080 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.321146 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.324582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.321240 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.340272 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.339741 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e1793d-openshift-default-dockercfg-4jbqs\"" Nov 03 01:31:14.421936 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.421901 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.421991 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422026 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422051 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422076 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r848b\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422108 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422135 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422161 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422203 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422449 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422551 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422667 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422616 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.422724 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422695 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.423017 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.422995 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.424375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.424350 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.424582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.424540 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.504759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.504694 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.504907 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.504882 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r848b\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.550786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.550746 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:14.600405 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:14.600369 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccfe377-1b71-48c3-8db5-010e66b56805" path="/var/lib/kubelet/pods/0ccfe377-1b71-48c3-8db5-010e66b56805/volumes" Nov 03 01:31:15.002773 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:31:15.002711 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5dde6d_0680_4998_892a_ecd7ef811939.slice/crio-0dee25fd531a68cc0c7feb039523ddb206c814d74d6ce236c1ce2314d975dcd4 WatchSource:0}: Error finding container 0dee25fd531a68cc0c7feb039523ddb206c814d74d6ce236c1ce2314d975dcd4: Status 404 returned error can't find the container with id 0dee25fd531a68cc0c7feb039523ddb206c814d74d6ce236c1ce2314d975dcd4 Nov 03 01:31:15.004875 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.004849 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:31:15.006005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.005967 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:15.006078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.006046 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:15.006112 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.006086 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:15.701684 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.701643 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerStarted","Data":"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e"} Nov 03 01:31:15.702112 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.701719 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerStarted","Data":"0dee25fd531a68cc0c7feb039523ddb206c814d74d6ce236c1ce2314d975dcd4"} Nov 03 01:31:15.826665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:15.826616 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podStartSLOduration=1.826602083 podStartE2EDuration="1.826602083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:15.825319596 +0000 UTC m=+811.404142183" watchObservedRunningTime="2026-11-03 01:31:15.826602083 +0000 UTC m=+811.405424670" Nov 03 01:31:16.551495 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:16.551456 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:16.556603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:16.556577 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:16.707051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:16.707023 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:16.708239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:16.708218 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:21.452137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.452099 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:21.452672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.452160 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:21.736301 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.736266 4992 generic.go:356] "Generic (PLEG): container finished" podID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerID="1194fb66238bc601fc5d722db79fc31bafd587ac5e45983d8c9ae4991f048553" exitCode=0 Nov 03 01:31:21.736476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.736342 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerDied","Data":"1194fb66238bc601fc5d722db79fc31bafd587ac5e45983d8c9ae4991f048553"} Nov 03 01:31:21.736811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.736794 4992 scope.go:122] "RemoveContainer" containerID="1194fb66238bc601fc5d722db79fc31bafd587ac5e45983d8c9ae4991f048553" Nov 03 01:31:21.738265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.738238 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:21.738326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.738303 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:21.738360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:21.738337 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:22.743543 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:22.743508 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerStarted","Data":"be5c4a08ff1867413e0e67a13b0f3ce37fd7bf8592eef9473bb33c2910793978"} Nov 03 01:31:23.503060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:23.503023 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:31:23.503339 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:23.503316 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:23.503393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:23.503368 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:24.503576 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:24.503534 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:24.504074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:24.503631 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:25.503196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:25.503158 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:25.503374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:25.503215 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:26.503851 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:26.503815 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:26.504260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:26.503869 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:27.503224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:27.503186 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 01:31:27.503427 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:27.503253 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 01:31:28.507509 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:28.507478 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:31:28.507980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:28.507853 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:31:28.508519 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:28.508498 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 01:31:32.152419 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:31:32.152379 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:31:43.481985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.481949 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 01:31:43.485199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.485181 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.540635 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.540600 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 01:31:43.573798 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.573764 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.573964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.573806 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.573964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.573831 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.573964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.573904 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8kv\" (UniqueName: \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.590262 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.590226 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e1793d-serving-cert\"" Nov 03 01:31:43.674921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.674881 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.675126 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.674932 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.675126 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.674953 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.675126 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.674975 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8kv\" (UniqueName: \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.675126 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:31:43.675100 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-serving-cert: secret "maas-api-e2e-worker-w1-e1793d-serving-cert" not found Nov 03 01:31:43.675341 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:31:43.675176 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls podName:8d0729db-abdb-4e14-9d5f-cc2efc1ef72b nodeName:}" failed. No retries permitted until 2026-11-03 01:31:44.175154721 +0000 UTC m=+839.753977303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls") pod "maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" (UID: "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b") : secret "maas-api-e2e-worker-w1-e1793d-serving-cert" not found Nov 03 01:31:43.675704 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.675680 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.677503 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.677480 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:43.741789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:43.741707 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8kv\" (UniqueName: \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:44.180782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:44.180743 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:44.183251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:44.183222 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:44.398127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:44.398089 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:44.769784 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:31:44.769750 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0729db_abdb_4e14_9d5f_cc2efc1ef72b.slice/crio-6594f6907f79715243cd5f41132dfebd1716eeaed31092c47bf77c4a3dba609f WatchSource:0}: Error finding container 6594f6907f79715243cd5f41132dfebd1716eeaed31092c47bf77c4a3dba609f: Status 404 returned error can't find the container with id 6594f6907f79715243cd5f41132dfebd1716eeaed31092c47bf77c4a3dba609f Nov 03 01:31:44.772316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:44.772291 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 01:31:44.893207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:44.893167 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" event={"ID":"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b","Type":"ContainerStarted","Data":"6594f6907f79715243cd5f41132dfebd1716eeaed31092c47bf77c4a3dba609f"} Nov 03 01:31:45.899966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:45.899929 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" event={"ID":"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b","Type":"ContainerStarted","Data":"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5"} Nov 03 01:31:45.900358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:45.900009 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:46.058071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:46.057968 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" podStartSLOduration=2.15373159 podStartE2EDuration="3.057954004s" totalImagesPullingTime="904.222414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:46.057867495 +0000 UTC m=+841.636690079" watchObservedRunningTime="2026-11-03 01:31:46.057954004 +0000 UTC m=+841.636776570" Nov 03 01:31:51.911225 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:51.911190 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 01:31:57.965176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.965138 4992 generic.go:356] "Generic (PLEG): container finished" podID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerID="7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e" exitCode=0 Nov 03 01:31:57.965648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.965210 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerDied","Data":"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e"} Nov 03 01:31:57.965648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.965638 4992 scope.go:122] "RemoveContainer" containerID="7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e" Nov 03 01:31:57.966814 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.966787 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:57.966878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.966849 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:57.966917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:57.966880 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:31:58.992819 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:58.992789 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerStarted","Data":"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c"} Nov 03 01:31:59.551303 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:59.551271 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:31:59.553719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:59.552917 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 01:31:59.553719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:31:59.553358 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 01:32:00.551186 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.551146 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 01:32:00.551649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.551203 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 01:32:00.653997 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.653954 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 01:32:00.658439 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.658406 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.711205 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.711164 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 01:32:00.723171 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.723135 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"openshift-service-ca.crt\"" Nov 03 01:32:00.760797 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.760729 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"kube-root-ca.crt\"" Nov 03 01:32:00.760797 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.760729 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"default-dockercfg-m5ctc\"" Nov 03 01:32:00.761077 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.760954 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"simulator-w0-b6fb9b-kserve-self-signed-certs\"" Nov 03 01:32:00.807346 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807248 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.807346 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807286 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpqz\" (UniqueName: \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.807346 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807310 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.807680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807358 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.807680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807373 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.807680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.807440 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908179 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908226 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908245 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpqz\" (UniqueName: \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908273 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908322 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908345 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908691 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908790 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.908893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.908874 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.911297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.911265 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.911423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.911405 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.979812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.974798 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpqz\" (UniqueName: \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\") pod \"simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:00.999625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:00.997784 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:01.388583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:01.386990 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 01:32:01.395778 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:32:01.395716 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85e1494f_7f5e_4886_88c4_1e68effbfbb1.slice/crio-0b9e2eac8d5f851daa98c3ff4fa9552baad89fedf24e444f452cd0886837f38f WatchSource:0}: Error finding container 0b9e2eac8d5f851daa98c3ff4fa9552baad89fedf24e444f452cd0886837f38f: Status 404 returned error can't find the container with id 0b9e2eac8d5f851daa98c3ff4fa9552baad89fedf24e444f452cd0886837f38f Nov 03 01:32:01.551807 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:01.551767 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 01:32:01.552303 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:01.551835 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 01:32:01.706979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:01.706939 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:32:02.016973 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:02.016879 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" event={"ID":"85e1494f-7f5e-4886-88c4-1e68effbfbb1","Type":"ContainerStarted","Data":"0b9e2eac8d5f851daa98c3ff4fa9552baad89fedf24e444f452cd0886837f38f"} Nov 03 01:32:02.151278 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:32:02.151239 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:32:02.551805 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:02.551768 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 01:32:02.551991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:02.551836 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 01:32:03.551569 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:03.551532 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 01:32:03.551760 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:03.551608 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 01:32:04.029881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:04.029843 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" event={"ID":"85e1494f-7f5e-4886-88c4-1e68effbfbb1","Type":"ContainerStarted","Data":"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96"} Nov 03 01:32:04.152028 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:04.151973 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podStartSLOduration=1.864829352 podStartE2EDuration="4.151955842s" totalImagesPullingTime="2.28712649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:04.150231841 +0000 UTC m=+859.729054428" watchObservedRunningTime="2026-11-03 01:32:04.151955842 +0000 UTC m=+859.730778428" Nov 03 01:32:04.555552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:04.555522 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:32:04.556451 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:04.556432 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:32:04.998425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:04.998384 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:10.998932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:10.998886 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:11.012587 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:11.012532 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:11.078400 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:11.078366 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:32:32.151594 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:32:32.151537 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:32:37.414172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.414131 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 01:32:37.420878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.420851 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.473431 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.473392 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 01:32:37.482464 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482430 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48r5\" (UniqueName: \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.482658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482477 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.482658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482552 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.482658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482628 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.482838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482660 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.482838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.482756 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.523273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.523240 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"kube-root-ca.crt\"" Nov 03 01:32:37.523273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.523240 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"openshift-service-ca.crt\"" Nov 03 01:32:37.526814 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.526788 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"default-dockercfg-tk8nr\"" Nov 03 01:32:37.526872 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.526854 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"premium-w4-0bf34a-kserve-self-signed-certs\"" Nov 03 01:32:37.583271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583237 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48r5\" (UniqueName: \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583286 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583327 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583371 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583417 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583493 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583820 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583833 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.583912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.583873 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.585784 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.585759 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.586034 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.586018 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.646872 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.646789 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48r5\" (UniqueName: \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\") pod \"premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:37.733282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:37.733225 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:38.117686 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:32:38.117618 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod783a8dcd_1583_4cbe_9b69_c4a970b09d56.slice/crio-7447f38a5ba659d5f9cc7366af6e6e4553cfbffc6541779dd9cddca2b68e73bd WatchSource:0}: Error finding container 7447f38a5ba659d5f9cc7366af6e6e4553cfbffc6541779dd9cddca2b68e73bd: Status 404 returned error can't find the container with id 7447f38a5ba659d5f9cc7366af6e6e4553cfbffc6541779dd9cddca2b68e73bd Nov 03 01:32:38.119599 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:38.119575 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 01:32:38.224603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:38.224535 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" event={"ID":"783a8dcd-1583-4cbe-9b69-c4a970b09d56","Type":"ContainerStarted","Data":"009b3d54dfba0651e51e8b7a41736a390670913657ce805f27312c8ab2da5456"} Nov 03 01:32:38.224770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:38.224610 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" event={"ID":"783a8dcd-1583-4cbe-9b69-c4a970b09d56","Type":"ContainerStarted","Data":"7447f38a5ba659d5f9cc7366af6e6e4553cfbffc6541779dd9cddca2b68e73bd"} Nov 03 01:32:38.345571 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:38.345505 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podStartSLOduration=1.345487308 podStartE2EDuration="1.345487308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:38.344883748 +0000 UTC m=+893.923706335" watchObservedRunningTime="2026-11-03 01:32:38.345487308 +0000 UTC m=+893.924309896" Nov 03 01:32:38.734162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:38.734133 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:44.670830 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:44.670797 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:32:44.672625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:44.672602 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:32:47.734076 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:47.734036 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:47.746310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:47.746288 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:32:48.311810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:32:48.311784 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:33:02.151197 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:33:02.151152 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:33:03.946271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:03.946237 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 01:33:03.950062 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:03.950034 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.005082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.005045 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 01:33:04.054404 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.054371 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"premium-w0-b6fb9b-kserve-self-signed-certs\"" Nov 03 01:33:04.058930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.058900 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.058930 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.058931 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.059166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.059065 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lf9j\" (UniqueName: \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.059166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.059099 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.059166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.059127 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.059325 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.059249 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.160780 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.160749 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lf9j\" (UniqueName: \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.160780 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.160787 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.160873 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161134 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161286 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161213 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161286 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161230 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161286 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161256 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161514 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161452 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.161607 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.161537 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.163445 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.163423 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.163794 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.163773 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.226316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.226238 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lf9j\" (UniqueName: \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\") pod \"premium-w0-b6fb9b-kserve-685b77794c-tlhcz\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.262374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.262105 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:04.661765 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:33:04.661734 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5729d0ae_797e_40ad_85d3_9600a13ade3f.slice/crio-92f62124b9afd2df09eb118176529ec9cb9a70d52f6a4830a43f1d36bd026a02 WatchSource:0}: Error finding container 92f62124b9afd2df09eb118176529ec9cb9a70d52f6a4830a43f1d36bd026a02: Status 404 returned error can't find the container with id 92f62124b9afd2df09eb118176529ec9cb9a70d52f6a4830a43f1d36bd026a02 Nov 03 01:33:04.663233 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:04.663209 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 01:33:05.394603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:05.394570 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" event={"ID":"5729d0ae-797e-40ad-85d3-9600a13ade3f","Type":"ContainerStarted","Data":"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2"} Nov 03 01:33:05.395028 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:05.394615 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" event={"ID":"5729d0ae-797e-40ad-85d3-9600a13ade3f","Type":"ContainerStarted","Data":"92f62124b9afd2df09eb118176529ec9cb9a70d52f6a4830a43f1d36bd026a02"} Nov 03 01:33:05.517181 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:05.517124 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podStartSLOduration=2.517108282 podStartE2EDuration="2.517108282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:05.514866276 +0000 UTC m=+921.093688863" watchObservedRunningTime="2026-11-03 01:33:05.517108282 +0000 UTC m=+921.095930868" Nov 03 01:33:06.263023 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:06.262982 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:14.263180 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:14.263144 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:14.276248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:14.276220 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:14.451680 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:33:14.451645 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:33:32.150424 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:33:32.150388 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:34:02.151465 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:34:02.151423 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:34:32.149901 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:34:32.149863 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:34:44.729005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.728970 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 01:34:44.732719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.732699 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781641 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781598 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781660 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781731 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5f6\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781782 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781897 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.781986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.781921 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.782097 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.782030 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.782097 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.782082 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.782203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.782171 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.787238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.787222 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9da431-openshift-default-dockercfg-bq94q\"" Nov 03 01:34:44.841160 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.841130 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 01:34:44.883296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883260 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883446 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883302 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883446 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883382 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883446 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883418 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883457 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883481 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883522 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883587 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5f6\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883619 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.883919 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.883839 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.884055 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.884033 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.884114 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.884061 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.884213 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.884194 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.884271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.884236 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.885842 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.885815 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.885947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.885894 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.946575 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.946537 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:44.946763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:44.946740 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5f6\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:45.045633 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.045528 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:45.467650 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:34:45.467618 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc96c80b_b323_46da_8c66_3c5c57762655.slice/crio-4295d2170d4119bee719acbc4f1acb3abbe4a235d227919dd026b6351d035b28 WatchSource:0}: Error finding container 4295d2170d4119bee719acbc4f1acb3abbe4a235d227919dd026b6351d035b28: Status 404 returned error can't find the container with id 4295d2170d4119bee719acbc4f1acb3abbe4a235d227919dd026b6351d035b28 Nov 03 01:34:45.470040 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.470003 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:45.470109 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.470077 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:45.470146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.470107 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:34:45.470203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.470184 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 01:34:45.936163 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.936127 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" event={"ID":"fc96c80b-b323-46da-8c66-3c5c57762655","Type":"ContainerStarted","Data":"a1e0e349fa4b2252045e4a1e877e2325d6e59c82302729873f86892f70b38b97"} Nov 03 01:34:45.936722 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:45.936692 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" event={"ID":"fc96c80b-b323-46da-8c66-3c5c57762655","Type":"ContainerStarted","Data":"4295d2170d4119bee719acbc4f1acb3abbe4a235d227919dd026b6351d035b28"} Nov 03 01:34:46.046018 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:46.045978 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:46.047819 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:46.047795 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:46.047977 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:46.047849 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:46.061477 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:46.061432 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podStartSLOduration=2.061418327 podStartE2EDuration="2.061418327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:46.060236207 +0000 UTC m=+1021.639058793" watchObservedRunningTime="2026-11-03 01:34:46.061418327 +0000 UTC m=+1021.640240972" Nov 03 01:34:47.046834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:47.046782 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:47.047239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:47.046871 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:48.046487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:48.046442 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:48.046694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:48.046529 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:49.046921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:49.046885 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:49.047490 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:49.047456 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:50.046067 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:50.046019 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:50.046248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:50.046086 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:51.046665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:51.046626 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 01:34:51.047070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:51.046683 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 01:34:52.154516 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:52.154461 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:52.155005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:52.154817 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:34:52.156664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:34:52.156642 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 01:35:02.150493 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:35:02.150407 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:35:04.159629 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.159591 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:35:04.163472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.163450 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205307 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205351 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205432 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205501 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205576 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205631 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205879 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205667 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205879 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205765 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjscm\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.205879 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.205803 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.281737 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.281695 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:35:04.307059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307014 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307069 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307107 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307151 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307197 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjscm\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307227 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307273 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307306 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307360 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307524 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307589 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.307862 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307636 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.308003 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.307977 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.308425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.308402 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.310296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.310274 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.310402 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.310315 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.370539 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.370498 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.370730 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.370596 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjscm\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fvmnb\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.477303 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.477253 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:04.901981 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:35:04.901942 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15749875_6283_453a_a9d8_5c79ba723a83.slice/crio-13a523b23872d249daf0fc4a1cde5d421cf1a93037d43c57b9e8450afa7e639c WatchSource:0}: Error finding container 13a523b23872d249daf0fc4a1cde5d421cf1a93037d43c57b9e8450afa7e639c: Status 404 returned error can't find the container with id 13a523b23872d249daf0fc4a1cde5d421cf1a93037d43c57b9e8450afa7e639c Nov 03 01:35:04.903713 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.903689 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:35:04.904349 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.904319 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:04.904395 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.904384 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:04.904430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:04.904410 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:05.062799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.062763 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" event={"ID":"15749875-6283-453a-a9d8-5c79ba723a83","Type":"ContainerStarted","Data":"5e1bf043971502cad04f60bd95f1457d07fc1dfc04b742e3ec0a4e3bdb1589bd"} Nov 03 01:35:05.062966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.062824 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" event={"ID":"15749875-6283-453a-a9d8-5c79ba723a83","Type":"ContainerStarted","Data":"13a523b23872d249daf0fc4a1cde5d421cf1a93037d43c57b9e8450afa7e639c"} Nov 03 01:35:05.190132 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.190078 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podStartSLOduration=1.19006251 podStartE2EDuration="1.19006251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:05.18793435 +0000 UTC m=+1040.766756940" watchObservedRunningTime="2026-11-03 01:35:05.19006251 +0000 UTC m=+1040.768885097" Nov 03 01:35:05.478447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.478360 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:05.480223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.480191 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:05.480361 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:05.480255 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:06.480637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:06.480598 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:06.481530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:06.481347 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:07.483583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:07.482056 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:07.483583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:07.482134 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:08.479582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:08.479082 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:08.479582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:08.479175 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:09.481018 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:09.480975 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:09.481658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:09.481621 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:10.481042 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:10.481000 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:10.481678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:10.481640 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:11.478418 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:11.478374 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:11.478616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:11.478450 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:12.478147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:12.478105 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 03 01:35:12.478588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:12.478215 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 03 01:35:13.703896 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:13.703832 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:13.704409 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:13.704383 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:13.705764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:13.705743 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:35:32.149607 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:35:32.149570 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:35:44.724916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:44.724882 4992 scope.go:122] "RemoveContainer" containerID="2350ece34c6c5cc371c6267ccb0b008b1c755338bd03d8894ead2edc77d74ab6" Nov 03 01:35:46.413582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.412735 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 01:35:46.434582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.430840 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.550223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.550028 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 01:35:46.633223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633159 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633428 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633357 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633428 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633414 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633576 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633479 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633576 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633545 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633595 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633632 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633661 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh479\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.633714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.633708 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734484 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734590 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734641 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734695 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734753 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734783 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734817 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734842 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh479\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.734880 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.735260 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.735621 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.736627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.735931 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.737098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.736903 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.737695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.737659 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.741061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.741039 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.741242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.741128 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.826583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.815080 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:46.832323 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:46.832289 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh479\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:47.055487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:47.055400 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:47.630548 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:47.630513 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 01:35:47.635173 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:35:47.635143 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1bf105_6ac1_4d23_b5ce_904d17e86037.slice/crio-b78f7c39ee996213f4c6653b4af78927d441e9db970637c0319caf4cd1f73c98 WatchSource:0}: Error finding container b78f7c39ee996213f4c6653b4af78927d441e9db970637c0319caf4cd1f73c98: Status 404 returned error can't find the container with id b78f7c39ee996213f4c6653b4af78927d441e9db970637c0319caf4cd1f73c98 Nov 03 01:35:47.638244 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:47.638206 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:47.638354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:47.638274 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:47.638354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:47.638302 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:35:48.370581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:48.362309 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" event={"ID":"af1bf105-6ac1-4d23-b5ce-904d17e86037","Type":"ContainerStarted","Data":"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b"} Nov 03 01:35:48.370581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:48.362377 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" event={"ID":"af1bf105-6ac1-4d23-b5ce-904d17e86037","Type":"ContainerStarted","Data":"b78f7c39ee996213f4c6653b4af78927d441e9db970637c0319caf4cd1f73c98"} Nov 03 01:35:48.552655 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:48.552602 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podStartSLOduration=2.552590043 podStartE2EDuration="2.552590043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:48.552043129 +0000 UTC m=+1084.130865716" watchObservedRunningTime="2026-11-03 01:35:48.552590043 +0000 UTC m=+1084.131412626" Nov 03 01:35:49.055728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:49.055692 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:49.057008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:49.056975 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:49.057141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:49.057035 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:50.056088 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:50.056051 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:50.056480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:50.056107 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:51.056151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:51.056114 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:51.056540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:51.056174 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:52.056722 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:52.056681 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:52.057124 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:52.056739 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:53.055824 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:53.055788 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:53.056088 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:53.055844 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:54.056869 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:54.056825 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 01:35:54.057271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:54.056899 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 01:35:55.061526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:55.061495 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:55.061944 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:55.061890 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:55.062665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:55.062642 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 01:35:57.842061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.842024 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 01:35:57.846905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.846881 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.886831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.886796 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.887010 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.886835 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.887010 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.886858 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.887118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.887072 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.887189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.887171 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.887243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.887227 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjmm\" (UniqueName: \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.902758 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.902724 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 01:35:57.950764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.950714 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"kube-root-ca.crt\"" Nov 03 01:35:57.951101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.951071 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"openshift-service-ca.crt\"" Nov 03 01:35:57.952970 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.952943 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"distinct-w2-9da431-kserve-self-signed-certs\"" Nov 03 01:35:57.953076 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.952999 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"default-dockercfg-8v9fs\"" Nov 03 01:35:57.988058 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988016 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988073 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjmm\" (UniqueName: \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988172 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988197 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988226 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988329 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988512 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988482 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988735 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988709 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.988979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.988955 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:57.992851 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:57.992829 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:58.009996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:58.009963 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:58.054116 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:58.054082 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjmm\" (UniqueName: \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\") pod \"distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:58.165990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:58.165953 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:35:58.628790 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:35:58.628756 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a9b988d_9b20_4b02_9982_56335c023ab9.slice/crio-1e9096721611f69660c116b7075b922b01923013a56dec2f21b8fc9d1b52a2d3 WatchSource:0}: Error finding container 1e9096721611f69660c116b7075b922b01923013a56dec2f21b8fc9d1b52a2d3: Status 404 returned error can't find the container with id 1e9096721611f69660c116b7075b922b01923013a56dec2f21b8fc9d1b52a2d3 Nov 03 01:35:58.631114 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:58.631090 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 01:35:59.449960 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:59.449923 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" event={"ID":"1a9b988d-9b20-4b02-9982-56335c023ab9","Type":"ContainerStarted","Data":"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f"} Nov 03 01:35:59.450398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:59.449973 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" event={"ID":"1a9b988d-9b20-4b02-9982-56335c023ab9","Type":"ContainerStarted","Data":"1e9096721611f69660c116b7075b922b01923013a56dec2f21b8fc9d1b52a2d3"} Nov 03 01:35:59.577583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:35:59.577046 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podStartSLOduration=2.577025689 podStartE2EDuration="2.577025689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:59.575037063 +0000 UTC m=+1095.153859653" watchObservedRunningTime="2026-11-03 01:35:59.577025689 +0000 UTC m=+1095.155848278" Nov 03 01:36:00.175586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:00.171649 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:36:02.150629 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:36:02.150585 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:36:05.232589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.232529 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 01:36:05.236772 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.236749 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.291050 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.291015 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 01:36:05.296569 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.296527 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.296710 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.296684 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.296757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.296738 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7lt\" (UniqueName: \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.296900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.296874 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.297001 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.296979 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.297036 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.297026 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.340440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.340408 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"distinct-w0-b6fb9b-kserve-self-signed-certs\"" Nov 03 01:36:05.398202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398077 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398147 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398181 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398233 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398321 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398375 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7lt\" (UniqueName: \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398594 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398630 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.398901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.398836 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.400756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.400731 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.401019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.401002 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.462275 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.462219 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7lt\" (UniqueName: \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\") pod \"distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.553878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.553788 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:05.961287 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:36:05.961249 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7409933_d0c1_4bb6_b00c_e30749baf8ae.slice/crio-f79d95d22d33fc15a7d95767132c1edcd7f2405108a2fa8c17f6a176dee85362 WatchSource:0}: Error finding container f79d95d22d33fc15a7d95767132c1edcd7f2405108a2fa8c17f6a176dee85362: Status 404 returned error can't find the container with id f79d95d22d33fc15a7d95767132c1edcd7f2405108a2fa8c17f6a176dee85362 Nov 03 01:36:05.964200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:05.964168 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 01:36:06.488540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:06.488501 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" event={"ID":"a7409933-d0c1-4bb6-b00c-e30749baf8ae","Type":"ContainerStarted","Data":"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583"} Nov 03 01:36:06.488540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:06.488545 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" event={"ID":"a7409933-d0c1-4bb6-b00c-e30749baf8ae","Type":"ContainerStarted","Data":"f79d95d22d33fc15a7d95767132c1edcd7f2405108a2fa8c17f6a176dee85362"} Nov 03 01:36:06.554285 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:06.554251 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:06.609426 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:06.609371 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podStartSLOduration=1.609356859 podStartE2EDuration="1.609356859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:06.60886588 +0000 UTC m=+1102.187688467" watchObservedRunningTime="2026-11-03 01:36:06.609356859 +0000 UTC m=+1102.188179446" Nov 03 01:36:08.166961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:08.166920 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:36:08.179789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:08.179761 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:36:08.513009 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:08.512929 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:36:15.562582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:15.561529 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:15.635218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:15.634896 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:16.571257 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:16.571229 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:36:25.109300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.109220 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 01:36:25.113368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.113348 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.167467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.167433 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 01:36:25.196829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.196793 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.196994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.196836 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.196994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.196939 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59xw\" (UniqueName: \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.196994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.196981 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.197148 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.197125 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.197188 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.197156 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.216507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.216469 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"distinct-w4-0bf34a-kserve-self-signed-certs\"" Nov 03 01:36:25.298575 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.298759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298659 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.298759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298682 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.298759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298707 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.298759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298739 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.298968 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.298800 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59xw\" (UniqueName: \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.299302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.299276 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.299302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.299290 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.299461 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.299278 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.301253 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.301219 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.301820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.301797 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.362771 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.362688 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59xw\" (UniqueName: \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\") pod \"distinct-w4-0bf34a-kserve-d57964647-mxwkf\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.425774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.425733 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:25.812338 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:36:25.812306 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod210111a3_f7e5_4034_9011_3654ab84ff02.slice/crio-b2225696338756e324d918fa69887a54d326ffbe08dc5692333f026607008991 WatchSource:0}: Error finding container b2225696338756e324d918fa69887a54d326ffbe08dc5692333f026607008991: Status 404 returned error can't find the container with id b2225696338756e324d918fa69887a54d326ffbe08dc5692333f026607008991 Nov 03 01:36:25.814254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:25.814230 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 01:36:26.610214 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:26.610182 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" event={"ID":"210111a3-f7e5-4034-9011-3654ab84ff02","Type":"ContainerStarted","Data":"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d"} Nov 03 01:36:26.610662 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:26.610225 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" event={"ID":"210111a3-f7e5-4034-9011-3654ab84ff02","Type":"ContainerStarted","Data":"b2225696338756e324d918fa69887a54d326ffbe08dc5692333f026607008991"} Nov 03 01:36:26.732166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:26.732113 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podStartSLOduration=1.732098213 podStartE2EDuration="1.732098213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:26.730278704 +0000 UTC m=+1122.309101291" watchObservedRunningTime="2026-11-03 01:36:26.732098213 +0000 UTC m=+1122.310920822" Nov 03 01:36:27.425954 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:27.425919 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:32.151010 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:36:32.150971 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:36:35.426207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:35.426150 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:35.439072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:35.439039 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:35.670593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:35.670527 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:36:44.746366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:44.746331 4992 scope.go:122] "RemoveContainer" containerID="1e964726c3f06af51cf8f8ce74364b54f1d987f9f4f288878e2118bf14d07c6b" Nov 03 01:36:45.681198 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.681158 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:36:45.686449 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.686429 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.797026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.796985 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:36:45.830513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830474 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830723 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830520 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830723 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830695 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830738 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830851 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830809 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830872 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830909 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.830934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.830929 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.831020 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.831007 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932606 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932650 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932697 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932724 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932748 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.932918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932865 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.932933 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933188 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933157 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933188 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933173 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933221 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933238 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933277 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933324 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.933814 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.933796 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.935062 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.935045 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.935391 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.935369 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.996040 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.996004 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.996324 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.996290 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:45.999944 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:45.999909 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:46.427630 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:36:46.427593 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd55298_7ed7_49ff_ad1a_38c66cae9992.slice/crio-d44b36956d1e1b9bdd2d271b7d7adbf5c937a18a72c1a867e85659c18e02c474 WatchSource:0}: Error finding container d44b36956d1e1b9bdd2d271b7d7adbf5c937a18a72c1a867e85659c18e02c474: Status 404 returned error can't find the container with id d44b36956d1e1b9bdd2d271b7d7adbf5c937a18a72c1a867e85659c18e02c474 Nov 03 01:36:46.428273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.428249 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:36:46.429987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.429955 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:46.430060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.430016 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:46.430060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.430047 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:36:46.722728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.722635 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" event={"ID":"dfd55298-7ed7-49ff-ad1a-38c66cae9992","Type":"ContainerStarted","Data":"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235"} Nov 03 01:36:46.722728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.722691 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" event={"ID":"dfd55298-7ed7-49ff-ad1a-38c66cae9992","Type":"ContainerStarted","Data":"d44b36956d1e1b9bdd2d271b7d7adbf5c937a18a72c1a867e85659c18e02c474"} Nov 03 01:36:46.848343 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:46.848292 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podStartSLOduration=1.84827758 podStartE2EDuration="1.84827758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:46.846514212 +0000 UTC m=+1142.425336811" watchObservedRunningTime="2026-11-03 01:36:46.84827758 +0000 UTC m=+1142.427100166" Nov 03 01:36:47.000664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:47.000584 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:47.002284 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:47.002262 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:47.002394 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:47.002302 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:48.001228 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:48.001183 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:48.001783 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:48.001269 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:49.001111 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:49.001076 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:49.001289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:49.001131 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:50.001243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:50.001204 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:50.001435 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:50.001297 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:51.000366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:51.000323 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:51.000588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:51.000382 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:52.001297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:52.001011 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:52.001297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:52.001065 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:53.001194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:53.001151 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 01:36:53.001373 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:53.001215 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 01:36:54.007074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:54.007038 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:54.007577 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:54.007421 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:36:54.008239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:36:54.008218 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:37:02.150075 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:37:02.150029 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:37:32.151324 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:37:32.151280 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:37:44.710678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:44.710643 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:37:44.713023 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:44.713002 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:37:47.727319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.727281 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 01:37:47.731231 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.731213 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.785619 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.785587 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 01:37:47.834087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.834052 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b6fb9b\"/\"distinct-2-w0-b6fb9b-kserve-self-signed-certs\"" Nov 03 01:37:47.858694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.858662 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.858852 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.858704 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.858916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.858868 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.858994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.858978 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.859065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.859043 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.859151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.859128 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjcd\" (UniqueName: \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960273 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960348 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960386 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960433 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjcd\" (UniqueName: \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960506 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960816 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960537 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960875 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960815 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960925 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960877 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.960925 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.960891 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.962793 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.962771 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:47.963178 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:47.963157 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:48.040638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:48.040553 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjcd\" (UniqueName: \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\") pod \"distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:48.043321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:48.043295 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:48.423192 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:37:48.423152 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcdb8b33_8fe5_4814_9a7e_02a8f0247d4b.slice/crio-38220b22adf8a5e11166939b8f64e817aa3859f248332e9f569e0be207206e47 WatchSource:0}: Error finding container 38220b22adf8a5e11166939b8f64e817aa3859f248332e9f569e0be207206e47: Status 404 returned error can't find the container with id 38220b22adf8a5e11166939b8f64e817aa3859f248332e9f569e0be207206e47 Nov 03 01:37:48.425544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:48.425520 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 01:37:49.135254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:49.135214 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" event={"ID":"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b","Type":"ContainerStarted","Data":"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a"} Nov 03 01:37:49.135696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:49.135280 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" event={"ID":"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b","Type":"ContainerStarted","Data":"38220b22adf8a5e11166939b8f64e817aa3859f248332e9f569e0be207206e47"} Nov 03 01:37:49.258309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:49.258249 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podStartSLOduration=2.258236217 podStartE2EDuration="2.258236217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:49.255420099 +0000 UTC m=+1204.834242686" watchObservedRunningTime="2026-11-03 01:37:49.258236217 +0000 UTC m=+1204.837058804" Nov 03 01:37:50.044187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:50.044155 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:58.043660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:58.043539 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:58.056577 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:58.056532 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:37:58.199480 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:37:58.199448 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:38:02.151389 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:38:02.151352 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:38:32.150892 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:38:32.150857 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:38:46.609089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.609053 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:38:46.613343 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.613319 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.722150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.722114 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:38:46.744413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744377 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.744611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744422 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.744691 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744632 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.744757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744693 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.744757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744745 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.744982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.744965 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.745056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.745040 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.745111 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.745068 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz5n\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.745194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.745171 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846091 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846136 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846154 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz5n\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846275 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846320 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846350 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846626 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846420 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846626 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846462 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846626 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846589 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846656 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846661 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846797 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.846920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.846893 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.847133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.847114 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.848477 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.848459 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.848843 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.848825 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.913493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.913467 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.913684 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.913534 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz5n\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:46.927381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:46.927356 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:47.356180 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.356140 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:38:47.360978 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.360939 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:47.361096 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.361073 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:47.361187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.361167 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:47.491167 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.491126 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" event={"ID":"3b136c78-c96b-4bfb-a152-bf9834f63459","Type":"ContainerStarted","Data":"f331c712d24220ad8ede2ac91bc6f581fe9f59ffe651230737712de91b237c9b"} Nov 03 01:38:47.491376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.491195 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" event={"ID":"3b136c78-c96b-4bfb-a152-bf9834f63459","Type":"ContainerStarted","Data":"ab77999a890edfce1b5b6fcfe0d64a690319bcc9a332b0929298beb0c81549df"} Nov 03 01:38:47.618610 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.618488 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podStartSLOduration=1.6184688839999999 podStartE2EDuration="1.618468884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:47.614109503 +0000 UTC m=+1263.192932090" watchObservedRunningTime="2026-11-03 01:38:47.618468884 +0000 UTC m=+1263.197291474" Nov 03 01:38:47.927868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:47.927834 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:48.938582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:48.937862 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:48.938582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:48.937934 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:49.928071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:49.928028 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:49.928365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:49.928332 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:50.928749 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:50.928708 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:50.929206 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:50.928815 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:51.927744 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:51.927708 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:51.927908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:51.927772 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:52.928726 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:52.928690 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:52.929201 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:52.928765 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:53.928143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:53.928101 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 01:38:53.928357 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:53.928173 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 01:38:54.817677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.817645 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:38:54.821937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.821902 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.892779 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.892747 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\"" Nov 03 01:38:54.927115 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927078 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927125 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927214 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927249 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927282 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927278 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnhb\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927416 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927360 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927455 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927416 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927456 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.927648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.927630 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:54.937915 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:54.937875 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:38:55.028332 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028301 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028373 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028397 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028432 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028469 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028500 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnhb\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028531 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028861 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028592 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028861 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028614 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028858 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.028966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.028935 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.029369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.029349 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.029503 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.029427 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.029838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.029813 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.030800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.030779 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.031384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.031356 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.092720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.092645 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.092720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.092659 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnhb\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.135742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.135705 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:55.562960 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:38:55.562914 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecd72e5_f59b_4ca5_9810_2bfd305208eb.slice/crio-d145c88eba182e77b60a560fa6d880bf14bf0d0c4f08fd324888a04cc45bf409 WatchSource:0}: Error finding container d145c88eba182e77b60a560fa6d880bf14bf0d0c4f08fd324888a04cc45bf409: Status 404 returned error can't find the container with id d145c88eba182e77b60a560fa6d880bf14bf0d0c4f08fd324888a04cc45bf409 Nov 03 01:38:55.566234 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.564200 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:38:55.566234 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.565002 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:55.566234 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.565088 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:55.566234 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.565131 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:38:55.968733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.968691 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:38:55.969260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:55.968751 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:38:56.027800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.027769 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:56.028147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.028126 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:56.028840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.028820 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:38:56.595070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.595031 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" event={"ID":"fecd72e5-f59b-4ca5-9810-2bfd305208eb","Type":"ContainerStarted","Data":"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55"} Nov 03 01:38:56.595264 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.595090 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" event={"ID":"fecd72e5-f59b-4ca5-9810-2bfd305208eb","Type":"ContainerStarted","Data":"d145c88eba182e77b60a560fa6d880bf14bf0d0c4f08fd324888a04cc45bf409"} Nov 03 01:38:56.720197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:56.720142 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podStartSLOduration=2.720126934 podStartE2EDuration="2.720126934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:56.718881556 +0000 UTC m=+1272.297704147" watchObservedRunningTime="2026-11-03 01:38:56.720126934 +0000 UTC m=+1272.298949518" Nov 03 01:38:57.136119 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:57.136088 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:38:57.138141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:57.138109 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:38:57.138277 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:57.138156 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:38:58.137085 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:58.137035 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:38:58.137516 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:58.137109 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:38:59.136482 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:59.136450 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:38:59.136650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:38:59.136498 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:39:00.136584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:00.136528 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:39:00.137001 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:00.136617 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:39:01.136974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:01.136932 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:39:01.137357 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:01.137001 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:39:02.136933 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:02.136886 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 01:39:02.137101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:02.136955 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 01:39:02.151964 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:39:02.151923 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:39:04.188583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:04.184702 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:04.188583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:04.184804 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:04.781583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:04.774928 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:39:04.781583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:04.776360 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:39:04.781881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:04.781791 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:39:09.959475 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:09.959437 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:39:09.963746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:09.963723 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.046854 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.046821 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.046880 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.046907 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.046936 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766bh\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.046974 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.047005 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.047053 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047391 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.047112 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.047391 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.047140 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.051256 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.051229 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:39:10.147853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.147818 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.147853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.147854 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.147877 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766bh\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.147923 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148061 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.147948 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148109 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148194 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148229 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148254 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148411 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148416 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148444 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148667 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148553 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.148882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.148864 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.150397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.150370 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.150766 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.150749 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.210609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.210516 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.210609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.210599 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766bh\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.277725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.277686 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:10.702125 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:39:10.702095 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26fe2a8_8291_46a8_aca6_ffcc8a4229af.slice/crio-d07215227a485cd6a199398ade5dc4e7108fd7a8e6fe4bda634e4c3a7616e752 WatchSource:0}: Error finding container d07215227a485cd6a199398ade5dc4e7108fd7a8e6fe4bda634e4c3a7616e752: Status 404 returned error can't find the container with id d07215227a485cd6a199398ade5dc4e7108fd7a8e6fe4bda634e4c3a7616e752 Nov 03 01:39:10.704743 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.704708 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:10.704813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.704780 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:10.704848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.704814 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:10.705013 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:10.704993 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:39:11.698991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:11.698930 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" event={"ID":"e26fe2a8-8291-46a8-aca6-ffcc8a4229af","Type":"ContainerStarted","Data":"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26"} Nov 03 01:39:11.699414 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:11.699009 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" event={"ID":"e26fe2a8-8291-46a8-aca6-ffcc8a4229af","Type":"ContainerStarted","Data":"d07215227a485cd6a199398ade5dc4e7108fd7a8e6fe4bda634e4c3a7616e752"} Nov 03 01:39:11.896568 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:11.896485 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podStartSLOduration=2.896468386 podStartE2EDuration="2.896468386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:11.893787687 +0000 UTC m=+1287.472610275" watchObservedRunningTime="2026-11-03 01:39:11.896468386 +0000 UTC m=+1287.475290971" Nov 03 01:39:12.278469 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:12.278434 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:12.280976 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:12.280947 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:12.281165 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:12.281141 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:13.285713 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:13.285663 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:13.286340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:13.286304 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:14.285141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:14.285102 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:14.285446 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:14.285415 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:15.284586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:15.279927 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:15.284586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:15.279987 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:16.282583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:16.281991 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:16.282583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:16.282056 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:17.279091 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:17.279036 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 01:39:17.279616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:17.279107 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 01:39:19.327752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:19.327702 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:19.328370 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:19.327762 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:19.721551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:19.721515 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:19.722021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:19.721995 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:19.722914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:19.722897 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:39:23.319235 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.319201 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:39:23.324616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.324592 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.429671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.429639 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:39:23.481168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481129 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481378 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481183 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481378 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481207 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481427 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481624 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481601 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481665 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481727 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481847 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481822 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.481980 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.481866 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msc6z\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.583676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.583528 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.583992 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.583915 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584166 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msc6z\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584317 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584372 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584399 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584455 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584700 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584584 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.584700 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.584641 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.585674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.585207 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.585674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.585499 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.586160 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.586137 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.586261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.586237 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.587042 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.587020 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.588822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.588798 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.589645 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.589623 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.650321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.650282 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.650504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.650407 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msc6z\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:23.938918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:23.938879 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:24.364197 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:39:24.364161 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f98310c_805b_4d63_aa97_cff472ad4856.slice/crio-fcc71227d33ff1e6c17aca5bc564753f817209f52d45c6882caf281befa8f66a WatchSource:0}: Error finding container fcc71227d33ff1e6c17aca5bc564753f817209f52d45c6882caf281befa8f66a: Status 404 returned error can't find the container with id fcc71227d33ff1e6c17aca5bc564753f817209f52d45c6882caf281befa8f66a Nov 03 01:39:24.366987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.366947 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:24.367063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.367037 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:24.367099 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.367078 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:24.367430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.367408 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:39:24.812118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.812069 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" event={"ID":"7f98310c-805b-4d63-aa97-cff472ad4856","Type":"ContainerStarted","Data":"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671"} Nov 03 01:39:24.812298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.812137 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" event={"ID":"7f98310c-805b-4d63-aa97-cff472ad4856","Type":"ContainerStarted","Data":"fcc71227d33ff1e6c17aca5bc564753f817209f52d45c6882caf281befa8f66a"} Nov 03 01:39:24.939828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.939790 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:24.947156 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:24.947106 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podStartSLOduration=1.947091132 podStartE2EDuration="1.947091132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:24.944165766 +0000 UTC m=+1300.522988353" watchObservedRunningTime="2026-11-03 01:39:24.947091132 +0000 UTC m=+1300.525913719" Nov 03 01:39:25.941399 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:25.941362 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:25.941817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:25.941419 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:26.940119 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:26.940083 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:26.940316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:26.940144 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:27.939307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:27.939271 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:27.939740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:27.939327 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:28.939466 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:28.939429 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:28.939914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:28.939485 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:29.939831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:29.939794 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:29.940241 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:29.939850 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:30.940209 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:30.940165 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 01:39:30.940643 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:30.940241 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 01:39:31.945152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:31.945120 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:31.945638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:31.945494 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:31.946314 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:31.946294 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:39:32.157913 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:39:32.157874 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:39:36.205327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:36.204629 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:39:36.504638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:36.502707 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:37.602638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:37.601970 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 01:39:37.619633 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:37.604502 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:39:37.747583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:37.726053 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:39:38.518683 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.518597 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:39:38.523270 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.523236 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.578157 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.578102 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:39:38.655529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655494 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr8k\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.655529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655541 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655645 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655684 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655750 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655855 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655887 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656083 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.655987 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.656354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.656157 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757205 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757163 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757267 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr8k\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757294 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757322 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757612 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757478 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757612 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757511 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757612 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757598 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757773 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757616 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.757773 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.757669 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.758088 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.758069 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.758296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.758268 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.758691 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.758666 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.758847 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.758739 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.758847 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.758771 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.760393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.760372 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.761086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.761065 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.820321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.820247 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.820321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.820254 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr8k\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:38.838344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:38.838309 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:39.271439 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:39:39.271405 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode664a08e_94c9_442b_b835_81ee0ea0908e.slice/crio-b0ee68e6ddafe9893109714deda0fbeb5d23972b4661773fbbc20ad5440fd503 WatchSource:0}: Error finding container b0ee68e6ddafe9893109714deda0fbeb5d23972b4661773fbbc20ad5440fd503: Status 404 returned error can't find the container with id b0ee68e6ddafe9893109714deda0fbeb5d23972b4661773fbbc20ad5440fd503 Nov 03 01:39:39.274082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.274044 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:39.274200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.274114 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:39.274200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.274142 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:39.274522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.274495 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:39:39.966176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.966131 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" event={"ID":"e664a08e-94c9-442b-b835-81ee0ea0908e","Type":"ContainerStarted","Data":"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be"} Nov 03 01:39:39.966711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:39.966203 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" event={"ID":"e664a08e-94c9-442b-b835-81ee0ea0908e","Type":"ContainerStarted","Data":"b0ee68e6ddafe9893109714deda0fbeb5d23972b4661773fbbc20ad5440fd503"} Nov 03 01:39:40.096436 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:40.096366 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podStartSLOduration=2.096344492 podStartE2EDuration="2.096344492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:40.091787245 +0000 UTC m=+1315.670609842" watchObservedRunningTime="2026-11-03 01:39:40.096344492 +0000 UTC m=+1315.675167080" Nov 03 01:39:40.839223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:40.839188 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:40.840510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:40.840487 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:40.840627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:40.840540 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:41.839625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:41.839537 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:41.840038 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:41.839641 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:42.839583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:42.839525 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:42.839799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:42.839609 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:43.839456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:43.839426 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:43.839649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:43.839487 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:44.839599 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:44.839534 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:44.839989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:44.839620 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:45.839299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:45.839254 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 01:39:45.839470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:45.839334 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 01:39:47.112315 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:47.112283 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:47.113097 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:47.113077 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:47.113894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:47.113875 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:39:55.329746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.329701 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:39:55.335199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.335175 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.392255 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.392002 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:39:55.469369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469325 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469446 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469512 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469547 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469731 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469657 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469731 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469687 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469731 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469707 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469744 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.469845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.469831 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gp6\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571270 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571233 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571280 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571305 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571326 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571383 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gp6\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571479 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571543 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571609 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571664 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571928 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571746 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.571928 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.571780 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.572051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.572033 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.572127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.572102 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.572336 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.572316 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.573826 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.573803 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.574075 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.574056 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.634174 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.634098 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.634341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.634317 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gp6\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:55.651043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:55.651013 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:56.081430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:56.081396 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:39:56.111762 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:39:56.111732 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba4342_0784_43f1_8840_b7b4400f5c04.slice/crio-733170f4d22c46785740e5e18caef6b6630a924ce03c17b65c87493c4533c5bc WatchSource:0}: Error finding container 733170f4d22c46785740e5e18caef6b6630a924ce03c17b65c87493c4533c5bc: Status 404 returned error can't find the container with id 733170f4d22c46785740e5e18caef6b6630a924ce03c17b65c87493c4533c5bc Nov 03 01:39:56.113991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:56.113956 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:56.114089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:56.114021 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:56.114089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:56.114054 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:39:57.095051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.095019 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" event={"ID":"52ba4342-0784-43f1-8840-b7b4400f5c04","Type":"ContainerStarted","Data":"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6"} Nov 03 01:39:57.095544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.095075 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" event={"ID":"52ba4342-0784-43f1-8840-b7b4400f5c04","Type":"ContainerStarted","Data":"733170f4d22c46785740e5e18caef6b6630a924ce03c17b65c87493c4533c5bc"} Nov 03 01:39:57.259052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.258998 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podStartSLOduration=2.258981809 podStartE2EDuration="2.258981809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:57.256104413 +0000 UTC m=+1332.834927001" watchObservedRunningTime="2026-11-03 01:39:57.258981809 +0000 UTC m=+1332.837804396" Nov 03 01:39:57.651912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.651877 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:39:57.653786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.653759 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:57.653903 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:57.653809 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:39:58.651633 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:58.651594 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:58.652039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:58.651656 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:39:59.652206 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:59.652167 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:39:59.652648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:39:59.652236 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:40:00.651840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:00.651803 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:40:00.652029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:00.651868 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:40:01.652114 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:01.652076 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:40:01.652512 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:01.652144 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:40:02.154324 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:40:02.154283 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:40:02.663995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:02.663703 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:40:02.664657 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:02.664612 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:40:03.565706 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:03.565654 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 01:40:03.656025 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:03.655984 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 01:40:03.656354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:03.656323 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 01:40:03.666752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:03.666680 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:40:03.736304 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:03.736246 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.080148 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.080097 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.229711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.206592 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:40:04.303769 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.303710 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.455827 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.454265 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.585869 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.585822 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.679584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.676338 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:04.741146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.740936 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 01:40:04.795584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.794478 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:40:04.795584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.794549 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:40:04.795584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:04.794609 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:40:05.091653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.086717 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.101585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.092395 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:40:05.253689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.249128 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.373504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.267203 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:40:05.373504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.278465 4992 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.373504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.325830 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 01:40:05.373504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.340069 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:40:05.854586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.851167 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:05.854586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.851242 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.882259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.882210 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:05.904584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.900804 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:40:05.906369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.906324 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 01:40:05.950584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.926815 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:05.950900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.950859 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.964593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.957499 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:05.964593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.957575 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.964593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.958851 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:05.996585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:05.986695 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:40:06.101584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.100727 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.301792 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.301737 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.411696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.351341 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.973296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.970663 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.973296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.971194 4992 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded" Nov 03 01:40:06.974547 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.974040 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:06.982586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.978681 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:06.982586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.980253 4992 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded" Nov 03 01:40:06.982586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.980356 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:06.982586 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:06.980387 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:07.036622 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.023467 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 01:40:07.118585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.117772 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:07.363658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.353295 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:07.363658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.355685 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 01:40:07.458544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.455072 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 01:40:07.502775 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.502717 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 01:40:07.615191 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.610912 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:40:07.661029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.643592 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 01:40:07.661029 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.649364 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 01:40:07.661753 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.661724 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 01:40:07.678299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:07.676760 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:40:08.024467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:08.024187 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:40:08.024467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:08.024283 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:08.700484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:08.700450 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:40:08.701397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:08.701367 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:40:08.702664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:08.702637 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:40:28.630583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:28.626652 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:28.653745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:28.653691 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:28.738753 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:28.738695 4992 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.354720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:29.354662 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.369720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:29.369652 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.630777 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:29.630666 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:29.687581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:29.684323 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:40:29.941590 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:29.936640 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 01:40:30.106596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.055550 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.119590 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.107290 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.376771 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.376655 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:40:30.377464 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.377438 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:40:30.674707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.674657 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 01:40:30.756190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.756159 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 01:40:30.861140 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:30.861109 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 01:40:32.150756 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:40:32.150710 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:40:49.469004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.468970 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:40:49.473086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.473062 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.528212 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.528178 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:40:49.575219 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.575184 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b9f22c1-serving-cert\"" Nov 03 01:40:49.585025 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.584973 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.585199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.585172 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz94x\" (UniqueName: \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.585345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.585321 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.585447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.585434 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.686961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.686918 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.687144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.687019 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz94x\" (UniqueName: \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.687144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.687087 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.687266 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:40:49.687174 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-serving-cert: secret "maas-api-e2e-mt-2b9f22c1-serving-cert" not found Nov 03 01:40:49.687266 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.687179 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.687266 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:40:49.687249 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls podName:9047fdf6-c073-4e8c-877d-911051bb235d nodeName:}" failed. No retries permitted until 2026-11-03 01:40:50.187230325 +0000 UTC m=+1385.766052904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls") pod "maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" (UID: "9047fdf6-c073-4e8c-877d-911051bb235d") : secret "maas-api-e2e-mt-2b9f22c1-serving-cert" not found Nov 03 01:40:49.687683 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.687656 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.689832 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.689811 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:49.752935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:49.752850 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz94x\" (UniqueName: \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:50.193417 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:50.193374 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:50.195757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:50.195733 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:50.386853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:50.386810 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:50.761463 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:40:50.761431 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9047fdf6_c073_4e8c_877d_911051bb235d.slice/crio-7e468059711ec92e6ec2c88610eccc602a0e3024a3dd7e45956e257691e6ee06 WatchSource:0}: Error finding container 7e468059711ec92e6ec2c88610eccc602a0e3024a3dd7e45956e257691e6ee06: Status 404 returned error can't find the container with id 7e468059711ec92e6ec2c88610eccc602a0e3024a3dd7e45956e257691e6ee06 Nov 03 01:40:50.763364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:50.763343 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:40:50.765011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:50.764989 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:40:51.511173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.511139 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" event={"ID":"9047fdf6-c073-4e8c-877d-911051bb235d","Type":"ContainerStarted","Data":"7e468059711ec92e6ec2c88610eccc602a0e3024a3dd7e45956e257691e6ee06"} Nov 03 01:40:51.757454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.757420 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 01:40:51.762510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.762439 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.825276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.825239 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 01:40:51.869043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.868983 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"default-dockercfg-c22xk\"" Nov 03 01:40:51.869043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.868984 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"distinct-w3-41aa5f-kserve-self-signed-certs\"" Nov 03 01:40:51.869254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.869053 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"kube-root-ca.crt\"" Nov 03 01:40:51.870859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.870841 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"openshift-service-ca.crt\"" Nov 03 01:40:51.918674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.918635 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.918912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.918765 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.918912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.918843 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.919039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.918931 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khht\" (UniqueName: \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.919039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.918969 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:51.919150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:51.919085 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.019917 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.019982 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020072 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020123 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020200 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khht\" (UniqueName: \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020237 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.020882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020829 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.021019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.020970 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.021137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.021115 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.030441 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.030409 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.037074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.037039 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.084412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.084373 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khht\" (UniqueName: \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\") pod \"distinct-w3-41aa5f-kserve-b85867c76-n6bj7\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.376394 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.376302 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:52.521957 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.521914 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" event={"ID":"9047fdf6-c073-4e8c-877d-911051bb235d","Type":"ContainerStarted","Data":"b01317d08ad3ad9cb002b63bddac424fb21aaf912d42a885c235d719772399e4"} Nov 03 01:40:52.522530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.522491 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:40:52.644466 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.644369 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" podStartSLOduration=2.590717927 podStartE2EDuration="3.64435697s" totalImagesPullingTime="1.053639043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:52.643727149 +0000 UTC m=+1388.222549735" watchObservedRunningTime="2026-11-03 01:40:52.64435697 +0000 UTC m=+1388.223179557" Nov 03 01:40:52.799619 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:40:52.799583 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca40087_ecfa_4848_bafc_71ec434da390.slice/crio-84b6a776521b7ce4bb855e83a21dc9fc91dd12e2ceca374eb0c01280a305780f WatchSource:0}: Error finding container 84b6a776521b7ce4bb855e83a21dc9fc91dd12e2ceca374eb0c01280a305780f: Status 404 returned error can't find the container with id 84b6a776521b7ce4bb855e83a21dc9fc91dd12e2ceca374eb0c01280a305780f Nov 03 01:40:52.802279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:52.802257 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 01:40:53.530386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:53.530347 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" event={"ID":"9ca40087-ecfa-4848-bafc-71ec434da390","Type":"ContainerStarted","Data":"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911"} Nov 03 01:40:53.530605 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:53.530394 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" event={"ID":"9ca40087-ecfa-4848-bafc-71ec434da390","Type":"ContainerStarted","Data":"84b6a776521b7ce4bb855e83a21dc9fc91dd12e2ceca374eb0c01280a305780f"} Nov 03 01:40:53.653593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:53.653491 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" podStartSLOduration=2.653472616 podStartE2EDuration="2.653472616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:53.651782663 +0000 UTC m=+1389.230605254" watchObservedRunningTime="2026-11-03 01:40:53.653472616 +0000 UTC m=+1389.232295205" Nov 03 01:40:54.379639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:54.378661 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:40:58.535831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:40:58.535800 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:41:02.151215 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:41:02.151175 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:41:02.377208 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:02.377176 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:41:02.390150 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:02.390122 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:41:02.646386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:02.646354 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 01:41:19.007374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.007340 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:41:19.011926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.011906 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.114607 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.114534 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.114781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.114677 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.114845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.114806 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.114952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.114933 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.115033 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.115011 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmckq\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.115073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.115054 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.115121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.115095 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.115274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.115247 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.115376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.115358 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.138890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.138858 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:41:19.216520 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216480 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216640 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216733 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216767 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216762 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmckq\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216791 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216834 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216902 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.216989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216983 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.217193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.216989 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.217193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.217066 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.217303 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.217272 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.217399 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.217373 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.217505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.217489 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.218947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.218928 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.219295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.219279 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.283177 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.283103 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.283314 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.283273 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmckq\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.325357 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.325325 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:19.753922 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:41:19.753893 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660224da_d543_4bea_85e0_c7afc039a780.slice/crio-284eda1b0b4957c4a39147a79b44de834115a9279774a426e34a5a48f594d495 WatchSource:0}: Error finding container 284eda1b0b4957c4a39147a79b44de834115a9279774a426e34a5a48f594d495: Status 404 returned error can't find the container with id 284eda1b0b4957c4a39147a79b44de834115a9279774a426e34a5a48f594d495 Nov 03 01:41:19.756676 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.756643 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:19.756765 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.756707 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:19.756765 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.756738 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:41:19.757084 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:19.757057 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:41:20.745298 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:20.745266 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" event={"ID":"660224da-d543-4bea-85e0-c7afc039a780","Type":"ContainerStarted","Data":"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6"} Nov 03 01:41:20.745709 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:20.745315 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" event={"ID":"660224da-d543-4bea-85e0-c7afc039a780","Type":"ContainerStarted","Data":"284eda1b0b4957c4a39147a79b44de834115a9279774a426e34a5a48f594d495"} Nov 03 01:41:20.926444 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:20.924550 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podStartSLOduration=2.924529291 podStartE2EDuration="2.924529291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:20.920876042 +0000 UTC m=+1416.499698630" watchObservedRunningTime="2026-11-03 01:41:20.924529291 +0000 UTC m=+1416.503351879" Nov 03 01:41:21.326073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:21.326028 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:21.327882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:21.327857 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:21.328007 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:21.327908 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:22.326129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:22.326089 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:22.326530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:22.326148 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:23.326448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:23.326410 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:23.326918 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:23.326471 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:24.325830 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:24.325794 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:24.326021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:24.325850 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:25.326399 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:25.326362 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:25.326908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:25.326424 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:26.325736 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:26.325698 4992 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 01:41:26.325910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:26.325754 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 01:41:27.330207 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:27.330177 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:27.330677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:27.330655 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:27.331281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:27.331265 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:41:29.001986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.001953 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 01:41:29.007203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.007184 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.060746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.060679 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 01:41:29.064406 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064374 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.064541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064414 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbk4\" (UniqueName: \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.064541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064443 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.064720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064694 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.064778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064733 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.064778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.064760 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.108717 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.108687 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41aa5f\"/\"distinct-2-w3-41aa5f-kserve-self-signed-certs\"" Nov 03 01:41:29.165989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.165956 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.165989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.165994 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbk4\" (UniqueName: \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166035 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166194 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166308 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166218 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166308 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166242 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166536 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166654 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166639 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.166712 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.166699 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.168806 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.168779 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.169314 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.169298 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.229058 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.229014 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbk4\" (UniqueName: \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\") pod \"distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.321130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.321039 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:29.707044 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:41:29.707013 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d9b1098_bb8d_4333_9c05_5a206aacb738.slice/crio-2213669c2a86996704c652b3877c4a262d974e759e3f5ece844d9669a3fb6482 WatchSource:0}: Error finding container 2213669c2a86996704c652b3877c4a262d974e759e3f5ece844d9669a3fb6482: Status 404 returned error can't find the container with id 2213669c2a86996704c652b3877c4a262d974e759e3f5ece844d9669a3fb6482 Nov 03 01:41:29.708632 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.708551 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 01:41:29.821467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.821435 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" event={"ID":"9d9b1098-bb8d-4333-9c05-5a206aacb738","Type":"ContainerStarted","Data":"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae"} Nov 03 01:41:29.821644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.821476 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" event={"ID":"9d9b1098-bb8d-4333-9c05-5a206aacb738","Type":"ContainerStarted","Data":"2213669c2a86996704c652b3877c4a262d974e759e3f5ece844d9669a3fb6482"} Nov 03 01:41:29.943773 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:29.943712 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" podStartSLOduration=1.943697995 podStartE2EDuration="1.943697995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:29.940680833 +0000 UTC m=+1425.519503420" watchObservedRunningTime="2026-11-03 01:41:29.943697995 +0000 UTC m=+1425.522520599" Nov 03 01:41:30.321914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:30.321880 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:32.151888 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:41:32.151848 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:41:39.321644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:39.321607 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:39.335277 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:39.335252 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:41:39.899691 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:41:39.899659 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 01:42:02.151237 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:42:02.151128 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:42:07.587694 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.587659 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:42:07.592074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.592051 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.648008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.647966 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:42:07.694962 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.694936 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-51bcf3-serving-cert\"" Nov 03 01:42:07.699652 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.699630 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.699762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.699736 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.699909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.699878 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl5hl\" (UniqueName: \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.700037 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.700023 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.801504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.801458 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.801768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.801550 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.801768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.801607 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl5hl\" (UniqueName: \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.801768 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:42:07.801647 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-serving-cert: secret "maas-api-e2e-api-a-51bcf3-serving-cert" not found Nov 03 01:42:07.801768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.801684 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.801768 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:42:07.801723 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls podName:a29e0910-e057-4b0b-83f8-02d504c13dcc nodeName:}" failed. No retries permitted until 2026-11-03 01:42:08.301694332 +0000 UTC m=+1463.880516910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls") pod "maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" (UID: "a29e0910-e057-4b0b-83f8-02d504c13dcc") : secret "maas-api-e2e-api-a-51bcf3-serving-cert" not found Nov 03 01:42:07.802473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.802453 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.804243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.804221 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:07.866487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:07.866413 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl5hl\" (UniqueName: \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:08.308387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:08.308353 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:08.310872 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:08.310851 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") pod \"maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:08.506026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:08.505991 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:08.893663 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:42:08.893617 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29e0910_e057_4b0b_83f8_02d504c13dcc.slice/crio-062e285e2759f01a07e5b86129a77e26fa7ff48f106421134c142448040e3d13 WatchSource:0}: Error finding container 062e285e2759f01a07e5b86129a77e26fa7ff48f106421134c142448040e3d13: Status 404 returned error can't find the container with id 062e285e2759f01a07e5b86129a77e26fa7ff48f106421134c142448040e3d13 Nov 03 01:42:08.897316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:08.897289 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:42:09.076021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:09.075990 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" event={"ID":"a29e0910-e057-4b0b-83f8-02d504c13dcc","Type":"ContainerStarted","Data":"062e285e2759f01a07e5b86129a77e26fa7ff48f106421134c142448040e3d13"} Nov 03 01:42:10.084071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:10.083965 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" event={"ID":"a29e0910-e057-4b0b-83f8-02d504c13dcc","Type":"ContainerStarted","Data":"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609"} Nov 03 01:42:10.084447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:10.084075 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:10.208751 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:10.208701 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" podStartSLOduration=2.329684905 podStartE2EDuration="3.208686837s" totalImagesPullingTime="879.001932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:10.205758119 +0000 UTC m=+1465.784580708" watchObservedRunningTime="2026-11-03 01:42:10.208686837 +0000 UTC m=+1465.787509423" Nov 03 01:42:16.096316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:16.096279 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:42:27.504836 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:27.504781 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-74bnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:42:27.505354 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:27.504860 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:42:32.150999 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:42:32.150961 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:42:32.685519 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.685481 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:42:32.689594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.689574 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.708654 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.708620 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4wz\" (UniqueName: \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.708799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.708681 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.708857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.708835 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.708983 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.708969 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.709019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.709007 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.709053 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.709027 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.745459 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.745422 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:42:32.792347 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.792314 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b9f22c1\"/\"kube-root-ca.crt\"" Nov 03 01:42:32.795590 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.795572 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b9f22c1\"/\"openshift-service-ca.crt\"" Nov 03 01:42:32.795672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.795584 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b9f22c1\"/\"e2e-lifecycle-model-2b9f22c1-kserve-self-signed-certs\"" Nov 03 01:42:32.795801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.795787 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b9f22c1\"/\"default-dockercfg-xzgwj\"" Nov 03 01:42:32.810790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810759 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.810914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810843 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.810914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810868 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.810914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810887 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.811070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810920 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4wz\" (UniqueName: \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.811070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.810975 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.811243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.811222 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.811313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.811243 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.811313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.811276 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.813296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.813273 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.813473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.813456 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:32.911774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:32.911741 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4wz\" (UniqueName: \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\") pod \"e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:33.002814 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:33.002725 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:33.391836 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:42:33.391788 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf462ddc_c83e_4aa3_8162_9b609cef4528.slice/crio-f451fe6c1d97a6f1bdae642f3f248e2d35d6279f9b2508717537f4ec89df9d59 WatchSource:0}: Error finding container f451fe6c1d97a6f1bdae642f3f248e2d35d6279f9b2508717537f4ec89df9d59: Status 404 returned error can't find the container with id f451fe6c1d97a6f1bdae642f3f248e2d35d6279f9b2508717537f4ec89df9d59 Nov 03 01:42:33.395309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:33.395284 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:42:34.295127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:34.295090 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" event={"ID":"cf462ddc-c83e-4aa3-8162-9b609cef4528","Type":"ContainerStarted","Data":"966f695bd150174faf4bc2a425076de6dcd68335ebc963d3c720e287b13b0ccd"} Nov 03 01:42:34.295316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:34.295137 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" event={"ID":"cf462ddc-c83e-4aa3-8162-9b609cef4528","Type":"ContainerStarted","Data":"f451fe6c1d97a6f1bdae642f3f248e2d35d6279f9b2508717537f4ec89df9d59"} Nov 03 01:42:34.416064 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:34.416007 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" podStartSLOduration=2.415993065 podStartE2EDuration="2.415993065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:34.414693495 +0000 UTC m=+1489.993516084" watchObservedRunningTime="2026-11-03 01:42:34.415993065 +0000 UTC m=+1489.994815653" Nov 03 01:42:35.003044 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:35.003008 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:43.003685 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:43.003648 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:43.016265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:43.016235 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:43.370366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:43.370294 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:42:44.755393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:44.755365 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:42:44.757366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:44.757345 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:42:52.391771 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.391732 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:42:52.397821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.397796 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.470581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.466934 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5e048b-openshift-default-dockercfg-76kqq\"" Nov 03 01:42:52.505584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.493347 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.511726 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.511806 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.511916 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.512051 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.512090 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.512114 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqjj\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.512143 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.530589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.512174 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.551776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.551737 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:42:52.613496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613465 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613505 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613531 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqjj\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613573 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613598 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613695 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613933 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613803 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613933 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613856 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.613933 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613914 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.614049 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.613952 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.614276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.614252 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.614635 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.614613 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.614894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.614872 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.615591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.615535 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.617226 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.617205 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.617733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.617712 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.677970 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.677930 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.678166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.678076 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqjj\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\") pod \"e2e-api-a-5e048b-openshift-default-6955d56565-sbft8\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:52.714048 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:52.714003 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:53.153360 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:42:53.153321 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb336bf7a_8c4a_45e9_b7a4_93505278f1a4.slice/crio-3a333d6c46aa9ff5d7d17e6b20926427bd06520a4a3a7d85d3adf57db3403f7e WatchSource:0}: Error finding container 3a333d6c46aa9ff5d7d17e6b20926427bd06520a4a3a7d85d3adf57db3403f7e: Status 404 returned error can't find the container with id 3a333d6c46aa9ff5d7d17e6b20926427bd06520a4a3a7d85d3adf57db3403f7e Nov 03 01:42:53.155734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.155704 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:42:53.156097 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.156068 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:53.156143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.156128 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:53.156176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.156154 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:42:53.427462 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.427426 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerStarted","Data":"a4fd4987984af279735c2f58f290f262c75ab40ee9bcfb65a5b72f29920f978c"} Nov 03 01:42:53.427974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.427478 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerStarted","Data":"3a333d6c46aa9ff5d7d17e6b20926427bd06520a4a3a7d85d3adf57db3403f7e"} Nov 03 01:42:53.554373 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.554308 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podStartSLOduration=1.554294963 podStartE2EDuration="1.554294963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:53.551323414 +0000 UTC m=+1509.130146001" watchObservedRunningTime="2026-11-03 01:42:53.554294963 +0000 UTC m=+1509.133117549" Nov 03 01:42:53.714839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:53.714760 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:54.720180 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:54.720150 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:55.440111 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:55.440083 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:42:55.441093 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:42:55.441072 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:43:02.150328 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:43:02.150287 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:43:10.471187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.471152 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:43:10.476728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.476695 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.525582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.524452 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:43:10.536802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.530406 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.536802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.536607 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41aa5f-openshift-default-dockercfg-g26tf\"" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623226 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623288 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623315 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623338 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623377 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5mv\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623404 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623459 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623490 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623530 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623553 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623621 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623662 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623710 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623821 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623852 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623887 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623913 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.629584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.623955 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mf7f\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.643802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.630791 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:43:10.643802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.632629 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:43:10.725977 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.725730 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726032 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726067 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726102 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726126 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726167 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mf7f\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726348 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726398 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726423 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726447 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726490 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5mv\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726521 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726539 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726627 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726667 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726711 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726740 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726771 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726808 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726810 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.726947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726859 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.727274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727075 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.727274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727074 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.727374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.726490 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.727374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727355 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.727653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727630 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.727794 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727772 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.727856 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.727843 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.728636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.728616 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.728901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.728883 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.729513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.729489 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.729647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.729629 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.791486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.791447 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5mv\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.791733 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.791688 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mf7f\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.791828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.791805 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:10.791895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.791836 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.840597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.840521 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:10.924580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:10.913038 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:11.313230 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.313144 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:43:11.338510 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:43:11.338476 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a064279_29a0_4c8e_9eda_8426698feba7.slice/crio-d9248dea81e0a10063028578386a4490a0610b2e18be3b0fb2400f9b3f77fe65 WatchSource:0}: Error finding container d9248dea81e0a10063028578386a4490a0610b2e18be3b0fb2400f9b3f77fe65: Status 404 returned error can't find the container with id d9248dea81e0a10063028578386a4490a0610b2e18be3b0fb2400f9b3f77fe65 Nov 03 01:43:11.341023 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.340990 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.341098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.341059 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.341098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.341087 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.358329 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:43:11.358299 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899a899b_ccfb_4739_9f75_fe33c2954301.slice/crio-c1fc9a7a09d8f420a5fe8e611c76f5b87ea5d2d5da88f0211a4e1f3163803555 WatchSource:0}: Error finding container c1fc9a7a09d8f420a5fe8e611c76f5b87ea5d2d5da88f0211a4e1f3163803555: Status 404 returned error can't find the container with id c1fc9a7a09d8f420a5fe8e611c76f5b87ea5d2d5da88f0211a4e1f3163803555 Nov 03 01:43:11.360248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.360212 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:43:11.361345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.361316 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.361507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.361392 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.361507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.361435 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:43:11.624297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.624186 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" event={"ID":"899a899b-ccfb-4739-9f75-fe33c2954301","Type":"ContainerStarted","Data":"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b"} Nov 03 01:43:11.624297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.624264 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" event={"ID":"899a899b-ccfb-4739-9f75-fe33c2954301","Type":"ContainerStarted","Data":"c1fc9a7a09d8f420a5fe8e611c76f5b87ea5d2d5da88f0211a4e1f3163803555"} Nov 03 01:43:11.626331 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.626300 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" event={"ID":"1a064279-29a0-4c8e-9eda-8426698feba7","Type":"ContainerStarted","Data":"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828"} Nov 03 01:43:11.626438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.626349 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" event={"ID":"1a064279-29a0-4c8e-9eda-8426698feba7","Type":"ContainerStarted","Data":"d9248dea81e0a10063028578386a4490a0610b2e18be3b0fb2400f9b3f77fe65"} Nov 03 01:43:11.766498 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.766439 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podStartSLOduration=1.7664243229999999 podStartE2EDuration="1.766424323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:11.763020438 +0000 UTC m=+1527.341843037" watchObservedRunningTime="2026-11-03 01:43:11.766424323 +0000 UTC m=+1527.345246909" Nov 03 01:43:11.840942 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.840909 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:11.906657 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.905782 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podStartSLOduration=1.9057428349999999 podStartE2EDuration="1.905742835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:11.90214823 +0000 UTC m=+1527.480970818" watchObservedRunningTime="2026-11-03 01:43:11.905742835 +0000 UTC m=+1527.484565421" Nov 03 01:43:11.921799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:11.921722 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:12.843155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:12.843118 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:12.843572 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:12.843178 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:12.923638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:12.923604 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:12.923803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:12.923667 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:13.841337 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:13.841294 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:13.841533 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:13.841365 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:13.921456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:13.921414 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:13.921878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:13.921485 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:14.841707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:14.841664 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:14.842021 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:14.841734 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:14.921416 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:14.921371 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:14.921643 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:14.921428 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:15.841592 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:15.841506 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:15.841868 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:15.841602 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:15.923157 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:15.922995 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:15.923157 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:15.923096 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:16.841810 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:16.841771 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:16.841993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:16.841828 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:16.923102 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:16.923064 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:16.923362 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:16.923334 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:17.841674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:17.841627 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 01:43:17.841858 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:17.841704 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 01:43:17.921914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:17.921875 4992 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 01:43:17.922082 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:17.921943 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 01:43:18.975390 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.975357 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:18.976062 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.976034 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:18.976890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.976862 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:18.977772 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.977750 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:18.978026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.978010 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:43:18.979985 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:18.979965 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:43:32.153103 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:43:32.153059 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:43:52.420538 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.420501 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:43:52.425895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.425871 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.464034 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.463999 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.464173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.464056 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgg26\" (UniqueName: \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.464218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.464202 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.464309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.464297 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.478925 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.478892 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:43:52.526803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.526766 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-012d31-serving-cert\"" Nov 03 01:43:52.565982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.565942 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.566193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.566041 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.566193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.566139 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.566193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.566176 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgg26\" (UniqueName: \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.566355 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:43:52.566215 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-serving-cert: secret "maas-api-e2e-api-b-012d31-serving-cert" not found Nov 03 01:43:52.566355 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:43:52.566316 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls podName:ebbf75c2-489e-4841-969e-925037922a63 nodeName:}" failed. No retries permitted until 2026-11-03 01:43:53.066292361 +0000 UTC m=+1568.645114940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls") pod "maas-api-e2e-api-b-012d31-75d649586f-t4fmn" (UID: "ebbf75c2-489e-4841-969e-925037922a63") : secret "maas-api-e2e-api-b-012d31-serving-cert" not found Nov 03 01:43:52.566839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.566808 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.568804 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.568783 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:52.629248 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:52.629211 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgg26\" (UniqueName: \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:53.075718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.075683 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:53.078190 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.078162 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") pod \"maas-api-e2e-api-b-012d31-75d649586f-t4fmn\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:53.340353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.340266 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:53.669175 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.669137 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7002-f3434a"] Nov 03 01:43:53.675306 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.675276 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:43:53.722285 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:43:53.722249 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbf75c2_489e_4841_969e_925037922a63.slice/crio-dec73bf4a532cc9a95be511a9421c94b7ae264a33ce5e4dd0f226ad1ca81f29b WatchSource:0}: Error finding container dec73bf4a532cc9a95be511a9421c94b7ae264a33ce5e4dd0f226ad1ca81f29b: Status 404 returned error can't find the container with id dec73bf4a532cc9a95be511a9421c94b7ae264a33ce5e4dd0f226ad1ca81f29b Nov 03 01:43:53.729623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.729550 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:43:53.741975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.741934 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-f3434a"] Nov 03 01:43:53.775755 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.775710 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\"" Nov 03 01:43:53.786911 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.786875 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcbh\" (UniqueName: \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\") pod \"test-curl-7002-f3434a\" (UID: \"03a3ad1f-7c65-4ed4-998d-bc097560e7eb\") " pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:43:53.888778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.888741 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcbh\" (UniqueName: \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\") pod \"test-curl-7002-f3434a\" (UID: \"03a3ad1f-7c65-4ed4-998d-bc097560e7eb\") " pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:43:53.951914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.951844 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcbh\" (UniqueName: \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\") pod \"test-curl-7002-f3434a\" (UID: \"03a3ad1f-7c65-4ed4-998d-bc097560e7eb\") " pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:43:53.966043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.966013 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" event={"ID":"ebbf75c2-489e-4841-969e-925037922a63","Type":"ContainerStarted","Data":"dec73bf4a532cc9a95be511a9421c94b7ae264a33ce5e4dd0f226ad1ca81f29b"} Nov 03 01:43:53.990267 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:53.990236 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:43:54.368844 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:43:54.368810 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03a3ad1f_7c65_4ed4_998d_bc097560e7eb.slice/crio-aee0202656bf7ea39fe1583c2c779f48a0f48ec451ead9b1d0b422dce1af85aa WatchSource:0}: Error finding container aee0202656bf7ea39fe1583c2c779f48a0f48ec451ead9b1d0b422dce1af85aa: Status 404 returned error can't find the container with id aee0202656bf7ea39fe1583c2c779f48a0f48ec451ead9b1d0b422dce1af85aa Nov 03 01:43:54.380132 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:54.373075 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-f3434a"] Nov 03 01:43:54.973928 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:54.973840 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" event={"ID":"ebbf75c2-489e-4841-969e-925037922a63","Type":"ContainerStarted","Data":"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8"} Nov 03 01:43:54.974355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:54.973974 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:43:54.975186 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:54.975156 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-f3434a" event={"ID":"03a3ad1f-7c65-4ed4-998d-bc097560e7eb","Type":"ContainerStarted","Data":"aee0202656bf7ea39fe1583c2c779f48a0f48ec451ead9b1d0b422dce1af85aa"} Nov 03 01:43:55.097669 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:55.097613 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" podStartSLOduration=2.152557695 podStartE2EDuration="3.097597724s" totalImagesPullingTime="945.040029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:55.094912508 +0000 UTC m=+1570.673735100" watchObservedRunningTime="2026-11-03 01:43:55.097597724 +0000 UTC m=+1570.676420311" Nov 03 01:43:56.021582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:56.018277 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-f3434a" event={"ID":"03a3ad1f-7c65-4ed4-998d-bc097560e7eb","Type":"ContainerStarted","Data":"f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477"} Nov 03 01:43:56.140171 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:56.140120 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7002-f3434a" podStartSLOduration=2.030471187 podStartE2EDuration="3.14010633s" totalImagesPullingTime="1.109635143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:56.137886573 +0000 UTC m=+1571.716709160" watchObservedRunningTime="2026-11-03 01:43:56.14010633 +0000 UTC m=+1571.718928917" Nov 03 01:43:57.692154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:57.692123 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7002-f3434a"] Nov 03 01:43:57.695300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:57.695275 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7002-f3434a"] Nov 03 01:43:58.033509 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:43:58.033390 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7002-f3434a" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" containerName="test-curl-7002-f3434a" containerID="cri-o://f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477" gracePeriod=2 Nov 03 01:43:58.087982 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:43:58.087924 4992 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-f3434a\" is forbidden: User \"system:node:ip-10-0-138-19.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-19.us-west-2.compute.internal' and this object" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:44:00.321727 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:00.321692 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:44:00.419789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:00.419755 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\" (UniqueName: \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\") pod \"03a3ad1f-7c65-4ed4-998d-bc097560e7eb\" (UID: \"03a3ad1f-7c65-4ed4-998d-bc097560e7eb\") " Nov 03 01:44:00.421899 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:00.421868 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh" pod "03a3ad1f-7c65-4ed4-998d-bc097560e7eb" (UID: "03a3ad1f-7c65-4ed4-998d-bc097560e7eb"). InnerVolumeSpecName "kube-api-access-nhcbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:00.521524 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:00.521483 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcbh\" (UniqueName: \"kubernetes.io/projected/03a3ad1f-7c65-4ed4-998d-bc097560e7eb-kube-api-access-nhcbh\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:44:00.596509 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:00.596421 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" path="/var/lib/kubelet/pods/03a3ad1f-7c65-4ed4-998d-bc097560e7eb/volumes" Nov 03 01:44:01.024030 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.023997 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:44:01.056937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.056901 4992 generic.go:356] "Generic (PLEG): container finished" podID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" containerID="f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477" exitCode=137 Nov 03 01:44:01.057131 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.056960 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-f3434a" Nov 03 01:44:01.057131 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.057004 4992 scope.go:122] "RemoveContainer" containerID="f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477" Nov 03 01:44:01.091860 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.091836 4992 scope.go:122] "RemoveContainer" containerID="f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477" Nov 03 01:44:01.092207 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:44:01.092179 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477\": container with ID starting with f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477 not found: ID does not exist" containerID="f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477" Nov 03 01:44:01.092300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:01.092214 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477"} err="failed to get container status \"f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477\": rpc error: code = NotFound desc = could not find container \"f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477\": container with ID starting with f34d2cf912dea12463be0629d094a4f86b62ebe807a18b6efb3fdb57e9775477 not found: ID does not exist" Nov 03 01:44:02.151836 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:44:02.151799 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:44:07.142465 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.142429 4992 generic.go:356] "Generic (PLEG): container finished" podID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerID="a4fd4987984af279735c2f58f290f262c75ab40ee9bcfb65a5b72f29920f978c" exitCode=0 Nov 03 01:44:07.143002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.142496 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerDied","Data":"a4fd4987984af279735c2f58f290f262c75ab40ee9bcfb65a5b72f29920f978c"} Nov 03 01:44:07.143002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.142933 4992 scope.go:122] "RemoveContainer" containerID="a4fd4987984af279735c2f58f290f262c75ab40ee9bcfb65a5b72f29920f978c" Nov 03 01:44:07.144502 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.144474 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:07.144597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.144539 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:07.144644 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:07.144605 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:08.151738 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:08.151707 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerStarted","Data":"6185f9bdee57d2567c58312a64e7ea8c55ff3eb1a32a7f865832d55d7f75a7ef"} Nov 03 01:44:08.714891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:08.714853 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:44:08.715774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:08.715748 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:44:08.715875 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:08.715804 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 01:44:09.715086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:09.715043 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:44:09.715656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:09.715108 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 01:44:10.440778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:10.440743 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:44:10.714696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:10.714616 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:44:10.714696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:10.714678 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 01:44:11.714974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:11.714938 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:44:11.715403 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:11.715016 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 01:44:12.714994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:12.714956 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 01:44:12.715386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:12.715010 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 01:44:13.719153 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:13.719119 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:44:13.720366 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:13.720342 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:44:16.491521 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.491475 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:44:16.492215 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.492196 4992 cpu_manager.go:414] "containerMap: removing container" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" containerName="test-curl-7002-f3434a" Nov 03 01:44:16.492215 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.492214 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" containerName="test-curl-7002-f3434a" Nov 03 01:44:16.492340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.492316 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a3ad1f-7c65-4ed4-998d-bc097560e7eb" containerName="test-curl-7002-f3434a" Nov 03 01:44:16.496475 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.496452 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.552960 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.552925 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4952b1-openshift-default-dockercfg-bzz9l\"" Nov 03 01:44:16.572535 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.572493 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.578595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.572840 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.579022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.578982 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.579179 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.579162 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.579324 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.579304 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pjx\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.579442 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.579428 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.583616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.579662 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.583974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.583945 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.584208 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.584189 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.679991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.679953 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.685463 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691240 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691307 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691339 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691413 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pjx\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691479 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691694 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.691893 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.692107 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.692687 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.692945 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.693973 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.696419 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.699596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.698866 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.702700 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.702657 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.705079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.703732 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.763744 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.763652 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pjx\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.769299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.769260 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\") pod \"e2e-api-b-4952b1-openshift-default-5d476677d9-s988c\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:16.816725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:16.816689 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:17.278678 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:44:17.278637 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb442232f_a4b4_459a_b851_63203d3d9bce.slice/crio-1ae966d70e824a1ac1fd8f9a2869e242808712c2e6712f8608d2010d4ef82656 WatchSource:0}: Error finding container 1ae966d70e824a1ac1fd8f9a2869e242808712c2e6712f8608d2010d4ef82656: Status 404 returned error can't find the container with id 1ae966d70e824a1ac1fd8f9a2869e242808712c2e6712f8608d2010d4ef82656 Nov 03 01:44:17.281522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:17.281486 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:44:17.281935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:17.281891 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:17.282025 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:17.281986 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:17.282079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:17.282033 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:44:18.244621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:18.244552 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerStarted","Data":"7de84c6a5fb82fa0e7c6d65a22ba99cf24c9d6a0cbcab74b8596c13189cd36b3"} Nov 03 01:44:18.245024 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:18.244641 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerStarted","Data":"1ae966d70e824a1ac1fd8f9a2869e242808712c2e6712f8608d2010d4ef82656"} Nov 03 01:44:18.421674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:18.421583 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podStartSLOduration=2.421567501 podStartE2EDuration="2.421567501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:18.42037353 +0000 UTC m=+1593.999196116" watchObservedRunningTime="2026-11-03 01:44:18.421567501 +0000 UTC m=+1594.000390079" Nov 03 01:44:18.818027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:18.817988 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:18.823675 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:18.823652 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:19.251707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:19.251674 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:19.252801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:44:19.252780 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:44:32.161178 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:44:32.161141 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:45:00.190259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.190223 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:00.196822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.196790 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.196990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.196789 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:00.201147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.201123 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:00.201295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.201228 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.205747 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.205722 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.263398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.263357 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.263635 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.263458 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqcd\" (UniqueName: \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.266315 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.266281 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.266433 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.266389 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9tv\" (UniqueName: \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.266433 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.266425 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl69n\" (UniqueName: \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.266523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.266504 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.268416 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.268393 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:00.284876 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.284494 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:00.289769 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.289743 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:00.334695 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.334661 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.430257 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.430372 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqcd\" (UniqueName: \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.430464 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.430522 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9tv\" (UniqueName: \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.431706 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.432633 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.430575 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl69n\" (UniqueName: \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.432878 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.447047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.446371 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.494019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.493990 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqcd\" (UniqueName: \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\") pod \"maas-api-key-cleanup-29894505-vts7t\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.494168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.494019 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl69n\" (UniqueName: \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\") pod \"maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.502842 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.502816 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9tv\" (UniqueName: \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\") pod \"maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.513753 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.513730 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:00.521850 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.521825 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:00.530720 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.530694 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:00.925920 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:00.925887 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19f2698_4028_45b3_b287_a67e78621f5d.slice/crio-e0bea687002c6ee84abad7a7f918feabe822baab1cc5654581c105058f7a1d1d WatchSource:0}: Error finding container e0bea687002c6ee84abad7a7f918feabe822baab1cc5654581c105058f7a1d1d: Status 404 returned error can't find the container with id e0bea687002c6ee84abad7a7f918feabe822baab1cc5654581c105058f7a1d1d Nov 03 01:45:00.929095 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.929074 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:00.976859 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:00.976492 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e86928_1766_4988_8eda_59870c93363f.slice/crio-abf5d3f9a56d8499f169e15c8b3db38ab47a141bc6299f0365d307f3ab3f1796 WatchSource:0}: Error finding container abf5d3f9a56d8499f169e15c8b3db38ab47a141bc6299f0365d307f3ab3f1796: Status 404 returned error can't find the container with id abf5d3f9a56d8499f169e15c8b3db38ab47a141bc6299f0365d307f3ab3f1796 Nov 03 01:45:00.980936 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:00.980910 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37eb1066_d59a_4319_8dc0_30addd747811.slice/crio-6105d3f72b44252ba23b097ea95656e67dc7639acb479a738993c2e07f24e18f WatchSource:0}: Error finding container 6105d3f72b44252ba23b097ea95656e67dc7639acb479a738993c2e07f24e18f: Status 404 returned error can't find the container with id 6105d3f72b44252ba23b097ea95656e67dc7639acb479a738993c2e07f24e18f Nov 03 01:45:00.982402 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.982296 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:00.985788 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:00.985766 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:01.575279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.575094 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerStarted","Data":"ba259ba185a82f150cb6270baa06e8a2de757afb148d734d82ae367d2c2effdf"} Nov 03 01:45:01.575808 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.575371 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerStarted","Data":"6105d3f72b44252ba23b097ea95656e67dc7639acb479a738993c2e07f24e18f"} Nov 03 01:45:01.579900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.579727 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerStarted","Data":"4e6f557d4d83315b5f1355d86e47936612de243c2cf7ccfb6f7a778039beb4b9"} Nov 03 01:45:01.579900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.579793 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerStarted","Data":"abf5d3f9a56d8499f169e15c8b3db38ab47a141bc6299f0365d307f3ab3f1796"} Nov 03 01:45:01.582263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.582229 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerStarted","Data":"d03f6cac07c27388968623d002c5fa683cbc58969b6aa843914f5defed68c17c"} Nov 03 01:45:01.582416 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.582300 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerStarted","Data":"e0bea687002c6ee84abad7a7f918feabe822baab1cc5654581c105058f7a1d1d"} Nov 03 01:45:01.696110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.696058 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" podStartSLOduration=1.696045188 podStartE2EDuration="1.696045188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:01.69360692 +0000 UTC m=+1637.272429505" watchObservedRunningTime="2026-11-03 01:45:01.696045188 +0000 UTC m=+1637.274867774" Nov 03 01:45:01.815113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:01.815012 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" podStartSLOduration=1.8149979539999999 podStartE2EDuration="1.814997954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:01.812180175 +0000 UTC m=+1637.391002762" watchObservedRunningTime="2026-11-03 01:45:01.814997954 +0000 UTC m=+1637.393820550" Nov 03 01:45:02.152082 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:02.152046 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:45:02.590153 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.590114 4992 generic.go:356] "Generic (PLEG): container finished" podID="f19f2698-4028-45b3-b287-a67e78621f5d" containerID="d03f6cac07c27388968623d002c5fa683cbc58969b6aa843914f5defed68c17c" exitCode=60 Nov 03 01:45:02.590673 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.590598 4992 scope.go:122] "RemoveContainer" containerID="d03f6cac07c27388968623d002c5fa683cbc58969b6aa843914f5defed68c17c" Nov 03 01:45:02.592146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.592122 4992 generic.go:356] "Generic (PLEG): container finished" podID="37eb1066-d59a-4319-8dc0-30addd747811" containerID="ba259ba185a82f150cb6270baa06e8a2de757afb148d734d82ae367d2c2effdf" exitCode=60 Nov 03 01:45:02.592537 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.592516 4992 scope.go:122] "RemoveContainer" containerID="ba259ba185a82f150cb6270baa06e8a2de757afb148d734d82ae367d2c2effdf" Nov 03 01:45:02.594486 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.594457 4992 generic.go:356] "Generic (PLEG): container finished" podID="c1e86928-1766-4988-8eda-59870c93363f" containerID="4e6f557d4d83315b5f1355d86e47936612de243c2cf7ccfb6f7a778039beb4b9" exitCode=60 Nov 03 01:45:02.594908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.594829 4992 scope.go:122] "RemoveContainer" containerID="4e6f557d4d83315b5f1355d86e47936612de243c2cf7ccfb6f7a778039beb4b9" Nov 03 01:45:02.597033 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.597008 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerDied","Data":"d03f6cac07c27388968623d002c5fa683cbc58969b6aa843914f5defed68c17c"} Nov 03 01:45:02.597127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.597048 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerDied","Data":"ba259ba185a82f150cb6270baa06e8a2de757afb148d734d82ae367d2c2effdf"} Nov 03 01:45:02.597127 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:02.597065 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerDied","Data":"4e6f557d4d83315b5f1355d86e47936612de243c2cf7ccfb6f7a778039beb4b9"} Nov 03 01:45:03.605649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.605613 4992 generic.go:356] "Generic (PLEG): container finished" podID="37eb1066-d59a-4319-8dc0-30addd747811" containerID="ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4" exitCode=60 Nov 03 01:45:03.606098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.605690 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerDied","Data":"ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4"} Nov 03 01:45:03.606098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.605750 4992 scope.go:122] "RemoveContainer" containerID="ba259ba185a82f150cb6270baa06e8a2de757afb148d734d82ae367d2c2effdf" Nov 03 01:45:03.606098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.606042 4992 scope.go:122] "RemoveContainer" containerID="ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4" Nov 03 01:45:03.606424 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:03.606403 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p_odh-ai-gateway-infra(37eb1066-d59a-4319-8dc0-30addd747811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" podUID="37eb1066-d59a-4319-8dc0-30addd747811" Nov 03 01:45:03.608884 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.608822 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerStarted","Data":"ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13"} Nov 03 01:45:03.613948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.613921 4992 generic.go:356] "Generic (PLEG): container finished" podID="f19f2698-4028-45b3-b287-a67e78621f5d" containerID="56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48" exitCode=60 Nov 03 01:45:03.614041 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.613969 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerDied","Data":"56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48"} Nov 03 01:45:03.614285 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.614272 4992 scope.go:122] "RemoveContainer" containerID="56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48" Nov 03 01:45:03.614578 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:03.614545 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-vts7t_odh-ai-gateway-infra(f19f2698-4028-45b3-b287-a67e78621f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" Nov 03 01:45:03.681981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:03.681955 4992 scope.go:122] "RemoveContainer" containerID="d03f6cac07c27388968623d002c5fa683cbc58969b6aa843914f5defed68c17c" Nov 03 01:45:04.636873 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.634429 4992 scope.go:122] "RemoveContainer" containerID="ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4" Nov 03 01:45:04.636873 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:04.634969 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p_odh-ai-gateway-infra(37eb1066-d59a-4319-8dc0-30addd747811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" podUID="37eb1066-d59a-4319-8dc0-30addd747811" Nov 03 01:45:04.637454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.637178 4992 generic.go:356] "Generic (PLEG): container finished" podID="c1e86928-1766-4988-8eda-59870c93363f" containerID="ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13" exitCode=60 Nov 03 01:45:04.637454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.637355 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerDied","Data":"ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13"} Nov 03 01:45:04.637454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.637409 4992 scope.go:122] "RemoveContainer" containerID="4e6f557d4d83315b5f1355d86e47936612de243c2cf7ccfb6f7a778039beb4b9" Nov 03 01:45:04.637926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.637902 4992 scope.go:122] "RemoveContainer" containerID="ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13" Nov 03 01:45:04.638328 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:04.638301 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz_odh-ai-gateway-infra(c1e86928-1766-4988-8eda-59870c93363f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" podUID="c1e86928-1766-4988-8eda-59870c93363f" Nov 03 01:45:04.640958 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:04.640935 4992 scope.go:122] "RemoveContainer" containerID="56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48" Nov 03 01:45:04.641309 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:04.641284 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-vts7t_odh-ai-gateway-infra(f19f2698-4028-45b3-b287-a67e78621f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" Nov 03 01:45:05.649226 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:05.649197 4992 scope.go:122] "RemoveContainer" containerID="ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13" Nov 03 01:45:05.649623 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:05.649551 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz_odh-ai-gateway-infra(c1e86928-1766-4988-8eda-59870c93363f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" podUID="c1e86928-1766-4988-8eda-59870c93363f" Nov 03 01:45:12.234210 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.234176 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:45:12.239738 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.239719 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.294197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.294169 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4e704-openshift-default-dockercfg-nt8mh\"" Nov 03 01:45:12.321790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.321754 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mm4\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.321991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.321863 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.321991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.321912 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.321991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.321955 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.322109 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.322076 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.322147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.322122 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.322291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.322159 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.322291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.322241 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.322291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.322269 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.350384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.350347 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:45:12.423789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.423750 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mm4\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.423979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.423811 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.423979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.423851 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.423979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.423890 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.423979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.423941 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424020 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424103 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424145 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424173 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424541 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424698 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424609 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424698 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424682 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.424803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.424705 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.425035 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.425009 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.426339 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.426313 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.426648 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.426631 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.486831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.486754 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mm4\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.486831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.486772 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-n59vn\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.554737 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.554701 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:12.981406 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:12.981376 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cfa2a8_1cf1_4899_be24_29b5c9a617b9.slice/crio-28517067756a79637fb68c3331ce853cc93b59d52c529594ab871200e120537f WatchSource:0}: Error finding container 28517067756a79637fb68c3331ce853cc93b59d52c529594ab871200e120537f: Status 404 returned error can't find the container with id 28517067756a79637fb68c3331ce853cc93b59d52c529594ab871200e120537f Nov 03 01:45:12.984001 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.983966 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:12.984091 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.984031 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:12.984091 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.984062 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:12.984745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:12.984718 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:45:13.721023 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:13.720975 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" event={"ID":"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9","Type":"ContainerStarted","Data":"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8"} Nov 03 01:45:13.721498 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:13.721080 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" event={"ID":"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9","Type":"ContainerStarted","Data":"28517067756a79637fb68c3331ce853cc93b59d52c529594ab871200e120537f"} Nov 03 01:45:13.856378 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:13.856319 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podStartSLOduration=1.856304654 podStartE2EDuration="1.856304654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:13.852360538 +0000 UTC m=+1649.431183125" watchObservedRunningTime="2026-11-03 01:45:13.856304654 +0000 UTC m=+1649.435127242" Nov 03 01:45:14.555770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.555736 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:14.557597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.557551 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:14.557750 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.557630 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 01:45:14.593514 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.593480 4992 scope.go:122] "RemoveContainer" containerID="ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4" Nov 03 01:45:14.598293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.598267 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:45:14.598501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.598465 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" podUID="9047fdf6-c073-4e8c-877d-911051bb235d" containerName="maas-api" containerID="cri-o://b01317d08ad3ad9cb002b63bddac424fb21aaf912d42a885c235d719772399e4" gracePeriod=30 Nov 03 01:45:14.732318 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.732284 4992 generic.go:356] "Generic (PLEG): container finished" podID="9047fdf6-c073-4e8c-877d-911051bb235d" containerID="b01317d08ad3ad9cb002b63bddac424fb21aaf912d42a885c235d719772399e4" exitCode=0 Nov 03 01:45:14.732719 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:14.732369 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" event={"ID":"9047fdf6-c073-4e8c-877d-911051bb235d","Type":"ContainerDied","Data":"b01317d08ad3ad9cb002b63bddac424fb21aaf912d42a885c235d719772399e4"} Nov 03 01:45:15.026470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.026184 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:45:15.092521 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.092482 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") pod \"9047fdf6-c073-4e8c-877d-911051bb235d\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " Nov 03 01:45:15.092869 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.092853 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\") pod \"9047fdf6-c073-4e8c-877d-911051bb235d\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " Nov 03 01:45:15.093068 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.093050 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\" (UniqueName: \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\") pod \"9047fdf6-c073-4e8c-877d-911051bb235d\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " Nov 03 01:45:15.093217 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.093206 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\") pod \"9047fdf6-c073-4e8c-877d-911051bb235d\" (UID: \"9047fdf6-c073-4e8c-877d-911051bb235d\") " Nov 03 01:45:15.094174 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.094142 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca" pod "9047fdf6-c073-4e8c-877d-911051bb235d" (UID: "9047fdf6-c073-4e8c-877d-911051bb235d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:15.095291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.095269 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls" pod "9047fdf6-c073-4e8c-877d-911051bb235d" (UID: "9047fdf6-c073-4e8c-877d-911051bb235d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:15.096013 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.095993 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs" pod "9047fdf6-c073-4e8c-877d-911051bb235d" (UID: "9047fdf6-c073-4e8c-877d-911051bb235d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:15.096253 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.096226 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x" pod "9047fdf6-c073-4e8c-877d-911051bb235d" (UID: "9047fdf6-c073-4e8c-877d-911051bb235d"). InnerVolumeSpecName "kube-api-access-nz94x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:15.201682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.197858 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz94x\" (UniqueName: \"kubernetes.io/projected/9047fdf6-c073-4e8c-877d-911051bb235d-kube-api-access-nz94x\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:15.201682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.197900 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9047fdf6-c073-4e8c-877d-911051bb235d-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:15.201682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.197918 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:15.201682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.197934 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9047fdf6-c073-4e8c-877d-911051bb235d-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:15.555067 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.555022 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:15.555272 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.555164 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 01:45:15.588319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.588287 4992 scope.go:122] "RemoveContainer" containerID="ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13" Nov 03 01:45:15.749157 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.749124 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" Nov 03 01:45:15.750882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.749144 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg" event={"ID":"9047fdf6-c073-4e8c-877d-911051bb235d","Type":"ContainerDied","Data":"7e468059711ec92e6ec2c88610eccc602a0e3024a3dd7e45956e257691e6ee06"} Nov 03 01:45:15.750882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.749280 4992 scope.go:122] "RemoveContainer" containerID="b01317d08ad3ad9cb002b63bddac424fb21aaf912d42a885c235d719772399e4" Nov 03 01:45:15.753750 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.753718 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerStarted","Data":"ef6285f998d78f4059ebe43322e76d357e43b5941e848193f1354f8af7050983"} Nov 03 01:45:15.921302 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.921275 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:45:15.989553 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:15.989514 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-b8hjg"] Nov 03 01:45:16.555912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.555875 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:16.556087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.555933 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 01:45:16.603859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.603825 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9047fdf6-c073-4e8c-877d-911051bb235d" path="/var/lib/kubelet/pods/9047fdf6-c073-4e8c-877d-911051bb235d/volumes" Nov 03 01:45:16.774070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.766349 4992 generic.go:356] "Generic (PLEG): container finished" podID="37eb1066-d59a-4319-8dc0-30addd747811" containerID="ef6285f998d78f4059ebe43322e76d357e43b5941e848193f1354f8af7050983" exitCode=60 Nov 03 01:45:16.774070 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.766511 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerDied","Data":"ef6285f998d78f4059ebe43322e76d357e43b5941e848193f1354f8af7050983"} Nov 03 01:45:16.774669 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.774101 4992 scope.go:122] "RemoveContainer" containerID="ac84fb7e812249f82065764d4755941a64fc91335e9b2bffecd6a7a91c4557b4" Nov 03 01:45:16.789585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.779999 4992 scope.go:122] "RemoveContainer" containerID="ef6285f998d78f4059ebe43322e76d357e43b5941e848193f1354f8af7050983" Nov 03 01:45:16.789585 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:16.789402 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p_odh-ai-gateway-infra(37eb1066-d59a-4319-8dc0-30addd747811)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" podUID="37eb1066-d59a-4319-8dc0-30addd747811" Nov 03 01:45:16.794639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:16.794602 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerStarted","Data":"397f710904d743c6c54e326ecbe82e1da6f5d6cf804e49fba41f9ec3e616d644"} Nov 03 01:45:17.055790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.055757 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:17.555989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.555954 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:17.556179 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.556011 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 01:45:17.587679 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.587642 4992 scope.go:122] "RemoveContainer" containerID="56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48" Nov 03 01:45:17.805653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.805620 4992 generic.go:356] "Generic (PLEG): container finished" podID="c1e86928-1766-4988-8eda-59870c93363f" containerID="397f710904d743c6c54e326ecbe82e1da6f5d6cf804e49fba41f9ec3e616d644" exitCode=60 Nov 03 01:45:17.806072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.805690 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerDied","Data":"397f710904d743c6c54e326ecbe82e1da6f5d6cf804e49fba41f9ec3e616d644"} Nov 03 01:45:17.806072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.805770 4992 scope.go:122] "RemoveContainer" containerID="ef60edb303a667d242ea40ce45198d6208d92705b0cefbe3d035c5b4168a5f13" Nov 03 01:45:17.806257 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:17.806240 4992 scope.go:122] "RemoveContainer" containerID="397f710904d743c6c54e326ecbe82e1da6f5d6cf804e49fba41f9ec3e616d644" Nov 03 01:45:17.806676 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:17.806653 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz_odh-ai-gateway-infra(c1e86928-1766-4988-8eda-59870c93363f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" podUID="c1e86928-1766-4988-8eda-59870c93363f" Nov 03 01:45:18.023333 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.023298 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:18.030995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.030966 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:18.141117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.141033 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\" (UniqueName: \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\") pod \"37eb1066-d59a-4319-8dc0-30addd747811\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " Nov 03 01:45:18.141117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.141079 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\") pod \"37eb1066-d59a-4319-8dc0-30addd747811\" (UID: \"37eb1066-d59a-4319-8dc0-30addd747811\") " Nov 03 01:45:18.141496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.141467 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca" pod "37eb1066-d59a-4319-8dc0-30addd747811" (UID: "37eb1066-d59a-4319-8dc0-30addd747811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:18.141715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.141691 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37eb1066-d59a-4319-8dc0-30addd747811-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:18.143309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.143286 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n" pod "37eb1066-d59a-4319-8dc0-30addd747811" (UID: "37eb1066-d59a-4319-8dc0-30addd747811"). InnerVolumeSpecName "kube-api-access-tl69n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:18.243050 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.243014 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl69n\" (UniqueName: \"kubernetes.io/projected/37eb1066-d59a-4319-8dc0-30addd747811-kube-api-access-tl69n\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:18.555788 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.555756 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 01:45:18.555958 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.555812 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 01:45:18.814036 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.813949 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" Nov 03 01:45:18.814036 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.813990 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p" event={"ID":"37eb1066-d59a-4319-8dc0-30addd747811","Type":"ContainerDied","Data":"6105d3f72b44252ba23b097ea95656e67dc7639acb479a738993c2e07f24e18f"} Nov 03 01:45:18.814537 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.814055 4992 scope.go:122] "RemoveContainer" containerID="ef6285f998d78f4059ebe43322e76d357e43b5941e848193f1354f8af7050983" Nov 03 01:45:18.818763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.818731 4992 generic.go:356] "Generic (PLEG): container finished" podID="f19f2698-4028-45b3-b287-a67e78621f5d" containerID="69f6afa05e57e0c26f98843e6279363039ec70dbb5544d4cbd24da5330863306" exitCode=60 Nov 03 01:45:18.818894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.818785 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerDied","Data":"69f6afa05e57e0c26f98843e6279363039ec70dbb5544d4cbd24da5330863306"} Nov 03 01:45:18.819311 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.819293 4992 scope.go:122] "RemoveContainer" containerID="69f6afa05e57e0c26f98843e6279363039ec70dbb5544d4cbd24da5330863306" Nov 03 01:45:18.819698 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:18.819673 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-vts7t_odh-ai-gateway-infra(f19f2698-4028-45b3-b287-a67e78621f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" Nov 03 01:45:18.825718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:18.825688 4992 scope.go:122] "RemoveContainer" containerID="56614cbee153bb6860b0d71e4bd487c1e85c15188d0ed527f6528b15c58a0c48" Nov 03 01:45:19.035214 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.035183 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:19.157572 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.157460 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\") pod \"c1e86928-1766-4988-8eda-59870c93363f\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " Nov 03 01:45:19.157769 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.157662 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\" (UniqueName: \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\") pod \"c1e86928-1766-4988-8eda-59870c93363f\" (UID: \"c1e86928-1766-4988-8eda-59870c93363f\") " Nov 03 01:45:19.157948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.157923 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca" pod "c1e86928-1766-4988-8eda-59870c93363f" (UID: "c1e86928-1766-4988-8eda-59870c93363f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:19.158180 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.158164 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1e86928-1766-4988-8eda-59870c93363f-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.159975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.159947 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv" pod "c1e86928-1766-4988-8eda-59870c93363f" (UID: "c1e86928-1766-4988-8eda-59870c93363f"). InnerVolumeSpecName "kube-api-access-fp9tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:19.242027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.241991 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:19.247499 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.247470 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-5e048b-29894505-dxz9p"] Nov 03 01:45:19.259333 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.259303 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp9tv\" (UniqueName: \"kubernetes.io/projected/c1e86928-1766-4988-8eda-59870c93363f-kube-api-access-fp9tv\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:19.559887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.559854 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:19.560310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.560270 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:19.561014 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.560999 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:45:19.836923 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.836835 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" Nov 03 01:45:19.837458 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.837426 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz" event={"ID":"c1e86928-1766-4988-8eda-59870c93363f","Type":"ContainerDied","Data":"abf5d3f9a56d8499f169e15c8b3db38ab47a141bc6299f0365d307f3ab3f1796"} Nov 03 01:45:19.837553 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:19.837492 4992 scope.go:122] "RemoveContainer" containerID="397f710904d743c6c54e326ecbe82e1da6f5d6cf804e49fba41f9ec3e616d644" Nov 03 01:45:20.010060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.010022 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:20.015613 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.015578 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-948470-29894505-k26qz"] Nov 03 01:45:20.021037 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.020801 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:20.185369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.185337 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:20.273677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.273640 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\") pod \"f19f2698-4028-45b3-b287-a67e78621f5d\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " Nov 03 01:45:20.273859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.273817 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\" (UniqueName: \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\") pod \"f19f2698-4028-45b3-b287-a67e78621f5d\" (UID: \"f19f2698-4028-45b3-b287-a67e78621f5d\") " Nov 03 01:45:20.274039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.274014 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca" pod "f19f2698-4028-45b3-b287-a67e78621f5d" (UID: "f19f2698-4028-45b3-b287-a67e78621f5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:20.274339 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.274326 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f19f2698-4028-45b3-b287-a67e78621f5d-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:20.276034 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.276010 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd" pod "f19f2698-4028-45b3-b287-a67e78621f5d" (UID: "f19f2698-4028-45b3-b287-a67e78621f5d"). InnerVolumeSpecName "kube-api-access-8gqcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:20.380882 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.380843 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqcd\" (UniqueName: \"kubernetes.io/projected/f19f2698-4028-45b3-b287-a67e78621f5d-kube-api-access-8gqcd\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:20.596346 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.596303 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37eb1066-d59a-4319-8dc0-30addd747811" path="/var/lib/kubelet/pods/37eb1066-d59a-4319-8dc0-30addd747811/volumes" Nov 03 01:45:20.596897 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.596873 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e86928-1766-4988-8eda-59870c93363f" path="/var/lib/kubelet/pods/c1e86928-1766-4988-8eda-59870c93363f/volumes" Nov 03 01:45:20.846412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.846309 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" event={"ID":"f19f2698-4028-45b3-b287-a67e78621f5d","Type":"ContainerDied","Data":"e0bea687002c6ee84abad7a7f918feabe822baab1cc5654581c105058f7a1d1d"} Nov 03 01:45:20.846412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.846383 4992 scope.go:122] "RemoveContainer" containerID="69f6afa05e57e0c26f98843e6279363039ec70dbb5544d4cbd24da5330863306" Nov 03 01:45:20.846412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:20.846335 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t" Nov 03 01:45:21.018148 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.018107 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:21.022963 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.022924 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-vts7t"] Nov 03 01:45:21.313374 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.313341 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:45:21.313975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.313960 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.313976 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.313985 4992 cpu_manager.go:414] "containerMap: removing container" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.313991 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314009 4992 cpu_manager.go:414] "containerMap: removing container" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314014 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314025 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314030 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314041 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314046 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314056 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9047fdf6-c073-4e8c-877d-911051bb235d" containerName="maas-api" Nov 03 01:45:21.314056 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314061 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9047fdf6-c073-4e8c-877d-911051bb235d" containerName="maas-api" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314075 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314080 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314089 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314093 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314102 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314107 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314192 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314201 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314207 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314216 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e86928-1766-4988-8eda-59870c93363f" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314221 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314227 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314235 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314241 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314248 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9047fdf6-c073-4e8c-877d-911051bb235d" containerName="maas-api" Nov 03 01:45:21.314398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.314255 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" containerName="cleanup" Nov 03 01:45:21.318570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.318539 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.373250 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.373210 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4952b1-serving-cert\"" Nov 03 01:45:21.374481 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.374452 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:45:21.395253 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.395213 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.395414 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.395351 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt28t\" (UniqueName: \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.395414 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.395379 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.395651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.395624 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.496907 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.496863 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt28t\" (UniqueName: \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.497079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.496925 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.497138 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:21.497089 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-serving-cert: secret "maas-api-e2e-api-b-4952b1-serving-cert" not found Nov 03 01:45:21.497186 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.497148 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.497186 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:21.497164 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls podName:d3e31f55-bd9e-4d32-8283-2fe1e13ba482 nodeName:}" failed. No retries permitted until 2026-11-03 01:45:21.997142545 +0000 UTC m=+1657.575965128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls") pod "maas-api-e2e-api-b-4952b1-779c849d69-l26p9" (UID: "d3e31f55-bd9e-4d32-8283-2fe1e13ba482") : secret "maas-api-e2e-api-b-4952b1-serving-cert" not found Nov 03 01:45:21.497603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.497306 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.497937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.497913 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.500490 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.500464 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:21.561942 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:21.561907 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt28t\" (UniqueName: \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:22.005193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.005151 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:22.007649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.007627 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") pod \"maas-api-e2e-api-b-4952b1-779c849d69-l26p9\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:22.239294 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.239250 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:22.596611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.596503 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19f2698-4028-45b3-b287-a67e78621f5d" path="/var/lib/kubelet/pods/f19f2698-4028-45b3-b287-a67e78621f5d/volumes" Nov 03 01:45:22.623965 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:22.623928 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e31f55_bd9e_4d32_8283_2fe1e13ba482.slice/crio-8a853f8c6b6fe3a84ed7c2ebbd6601f240756f3c151f493b7ec12ae5a4f7370a WatchSource:0}: Error finding container 8a853f8c6b6fe3a84ed7c2ebbd6601f240756f3c151f493b7ec12ae5a4f7370a: Status 404 returned error can't find the container with id 8a853f8c6b6fe3a84ed7c2ebbd6601f240756f3c151f493b7ec12ae5a4f7370a Nov 03 01:45:22.627417 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.627392 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:45:22.861442 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:22.861350 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" event={"ID":"d3e31f55-bd9e-4d32-8283-2fe1e13ba482","Type":"ContainerStarted","Data":"8a853f8c6b6fe3a84ed7c2ebbd6601f240756f3c151f493b7ec12ae5a4f7370a"} Nov 03 01:45:23.870059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:23.869968 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" event={"ID":"d3e31f55-bd9e-4d32-8283-2fe1e13ba482","Type":"ContainerStarted","Data":"a0bfe00829853eabd552c5b3be898956aea09e4f190d7b6d99df04cf4118e2ce"} Nov 03 01:45:23.870059 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:23.870039 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:23.993166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:23.993112 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" podStartSLOduration=2.0455132 podStartE2EDuration="2.993098352s" totalImagesPullingTime="947.585152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:23.990390178 +0000 UTC m=+1659.569212764" watchObservedRunningTime="2026-11-03 01:45:23.993098352 +0000 UTC m=+1659.571920939" Nov 03 01:45:25.987682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:25.987595 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:45:25.988154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:25.987814 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerName="maas-api" containerID="cri-o://f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609" gracePeriod=30 Nov 03 01:45:26.090400 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.090358 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.83:8443/health\": dial tcp 10.132.0.83:8443: connect: connection refused" Nov 03 01:45:26.327816 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.327780 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:45:26.380327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.380292 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") pod \"a29e0910-e057-4b0b-83f8-02d504c13dcc\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " Nov 03 01:45:26.380520 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.380433 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\") pod \"a29e0910-e057-4b0b-83f8-02d504c13dcc\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " Nov 03 01:45:26.380582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.380519 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\") pod \"a29e0910-e057-4b0b-83f8-02d504c13dcc\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " Nov 03 01:45:26.380582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.380549 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\" (UniqueName: \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\") pod \"a29e0910-e057-4b0b-83f8-02d504c13dcc\" (UID: \"a29e0910-e057-4b0b-83f8-02d504c13dcc\") " Nov 03 01:45:26.380919 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.380886 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca" pod "a29e0910-e057-4b0b-83f8-02d504c13dcc" (UID: "a29e0910-e057-4b0b-83f8-02d504c13dcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:26.381160 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.381138 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29e0910-e057-4b0b-83f8-02d504c13dcc-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.382812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.382786 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs" pod "a29e0910-e057-4b0b-83f8-02d504c13dcc" (UID: "a29e0910-e057-4b0b-83f8-02d504c13dcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:26.383136 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.383112 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl" pod "a29e0910-e057-4b0b-83f8-02d504c13dcc" (UID: "a29e0910-e057-4b0b-83f8-02d504c13dcc"). InnerVolumeSpecName "kube-api-access-wl5hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:26.383231 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.383195 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls" pod "a29e0910-e057-4b0b-83f8-02d504c13dcc" (UID: "a29e0910-e057-4b0b-83f8-02d504c13dcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:26.483247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.483208 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.483247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.483250 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl5hl\" (UniqueName: \"kubernetes.io/projected/a29e0910-e057-4b0b-83f8-02d504c13dcc-kube-api-access-wl5hl\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.483507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.483266 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29e0910-e057-4b0b-83f8-02d504c13dcc-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:26.892702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.892604 4992 generic.go:356] "Generic (PLEG): container finished" podID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerID="f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609" exitCode=0 Nov 03 01:45:26.892702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.892659 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" event={"ID":"a29e0910-e057-4b0b-83f8-02d504c13dcc","Type":"ContainerDied","Data":"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609"} Nov 03 01:45:26.892702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.892690 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" Nov 03 01:45:26.892975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.892711 4992 scope.go:122] "RemoveContainer" containerID="f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609" Nov 03 01:45:26.892975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.892698 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw" event={"ID":"a29e0910-e057-4b0b-83f8-02d504c13dcc","Type":"ContainerDied","Data":"062e285e2759f01a07e5b86129a77e26fa7ff48f106421134c142448040e3d13"} Nov 03 01:45:26.910135 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.910107 4992 scope.go:122] "RemoveContainer" containerID="f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609" Nov 03 01:45:26.910524 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:26.910495 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609\": container with ID starting with f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609 not found: ID does not exist" containerID="f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609" Nov 03 01:45:26.910610 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:26.910540 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609"} err="failed to get container status \"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609\": rpc error: code = NotFound desc = could not find container \"f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609\": container with ID starting with f3fd991074f65ed09a7abb2e425ba3c01c999dcdcff5ddf24aeaa7eb6db1c609 not found: ID does not exist" Nov 03 01:45:27.071993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:27.071952 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:45:27.118280 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:27.118244 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-51bcf3-56bdf7674f-nx5cw"] Nov 03 01:45:28.597169 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:28.597136 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" path="/var/lib/kubelet/pods/a29e0910-e057-4b0b-83f8-02d504c13dcc/volumes" Nov 03 01:45:29.883502 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:29.883475 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:45:32.150704 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:32.150664 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:45:46.078606 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.078495 4992 generic.go:356] "Generic (PLEG): container finished" podID="b442232f-a4b4-459a-b851-63203d3d9bce" containerID="7de84c6a5fb82fa0e7c6d65a22ba99cf24c9d6a0cbcab74b8596c13189cd36b3" exitCode=0 Nov 03 01:45:46.078606 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.078576 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerDied","Data":"7de84c6a5fb82fa0e7c6d65a22ba99cf24c9d6a0cbcab74b8596c13189cd36b3"} Nov 03 01:45:46.079104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.079081 4992 scope.go:122] "RemoveContainer" containerID="7de84c6a5fb82fa0e7c6d65a22ba99cf24c9d6a0cbcab74b8596c13189cd36b3" Nov 03 01:45:46.080828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.080795 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:46.080931 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.080867 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:46.080931 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:46.080906 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:47.099067 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:47.099027 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerStarted","Data":"5525e2a4f414bc3b32f25f66c01cb005bf826c9c424e5c3c37d857739f3b8a95"} Nov 03 01:45:47.818422 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:47.818377 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:45:47.819862 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:47.819835 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 01:45:47.820027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:47.819881 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 01:45:48.491541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.491501 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:45:48.492167 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.492150 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerName="maas-api" Nov 03 01:45:48.492215 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.492168 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerName="maas-api" Nov 03 01:45:48.492250 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.492224 4992 cpu_manager.go:414] "containerMap: removing container" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:48.492250 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.492230 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37eb1066-d59a-4319-8dc0-30addd747811" containerName="cleanup" Nov 03 01:45:48.492336 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.492326 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29e0910-e057-4b0b-83f8-02d504c13dcc" containerName="maas-api" Nov 03 01:45:48.495859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.495843 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.555440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.555062 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:45:48.601974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.601939 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-118857-serving-cert\"" Nov 03 01:45:48.630072 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.630024 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.631204 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.631085 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.631596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.631545 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq89n\" (UniqueName: \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.631859 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.631838 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.733987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.733939 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.734191 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.734335 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq89n\" (UniqueName: \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.734375 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.735343 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:48.735477 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-118857-serving-cert: secret "maas-api-e2e-api-b-118857-serving-cert" not found Nov 03 01:45:48.739031 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:48.735577 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls podName:e8041320-c248-4ebf-84fe-d8185015d25e nodeName:}" failed. No retries permitted until 2026-11-03 01:45:49.235533498 +0000 UTC m=+1684.814356087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls") pod "maas-api-e2e-api-b-118857-694b444f78-gvdt4" (UID: "e8041320-c248-4ebf-84fe-d8185015d25e") : secret "maas-api-e2e-api-b-118857-serving-cert" not found Nov 03 01:45:48.743729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.743007 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.800636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.800583 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq89n\" (UniqueName: \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:48.818731 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.818539 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 01:45:48.818731 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:48.818620 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 01:45:49.245328 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.245281 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:49.255974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.252797 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:45:49.256200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.256170 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") pod \"maas-api-e2e-api-b-118857-694b444f78-gvdt4\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:49.409791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.409750 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:49.815883 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:49.815841 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8041320_c248_4ebf_84fe_d8185015d25e.slice/crio-1ca753f4c8c8ccd10a76c0c1ddce13dbe92c0a4225dff52ff31165ae841909ab WatchSource:0}: Error finding container 1ca753f4c8c8ccd10a76c0c1ddce13dbe92c0a4225dff52ff31165ae841909ab: Status 404 returned error can't find the container with id 1ca753f4c8c8ccd10a76c0c1ddce13dbe92c0a4225dff52ff31165ae841909ab Nov 03 01:45:49.817818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.817789 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 01:45:49.817948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.817856 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 01:45:49.818139 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:49.818119 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:45:50.137761 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:50.133802 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" event={"ID":"e8041320-c248-4ebf-84fe-d8185015d25e","Type":"ContainerStarted","Data":"1ca753f4c8c8ccd10a76c0c1ddce13dbe92c0a4225dff52ff31165ae841909ab"} Nov 03 01:45:50.818252 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:50.818209 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 01:45:50.818630 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:50.818269 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 01:45:51.145260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:51.145173 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" event={"ID":"e8041320-c248-4ebf-84fe-d8185015d25e","Type":"ContainerStarted","Data":"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d"} Nov 03 01:45:51.145500 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:51.145483 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:51.272678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:51.269763 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" podStartSLOduration=2.282857803 podStartE2EDuration="3.269741087s" totalImagesPullingTime="986.883284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:51.268006715 +0000 UTC m=+1686.846829303" watchObservedRunningTime="2026-11-03 01:45:51.269741087 +0000 UTC m=+1686.848563676" Nov 03 01:45:51.823064 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:51.821109 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 01:45:51.823064 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:51.821183 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 01:45:52.823066 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:52.823029 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:45:52.824290 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:52.824253 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:45:53.159351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.159168 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:45:53.159535 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.159416 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" podUID="ebbf75c2-489e-4841-969e-925037922a63" containerName="maas-api" containerID="cri-o://e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8" gracePeriod=30 Nov 03 01:45:53.572800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.572766 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:45:53.582106 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.582061 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\") pod \"ebbf75c2-489e-4841-969e-925037922a63\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " Nov 03 01:45:53.582254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.582199 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") pod \"ebbf75c2-489e-4841-969e-925037922a63\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " Nov 03 01:45:53.582312 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.582252 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\" (UniqueName: \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\") pod \"ebbf75c2-489e-4841-969e-925037922a63\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " Nov 03 01:45:53.584405 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.584369 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls" pod "ebbf75c2-489e-4841-969e-925037922a63" (UID: "ebbf75c2-489e-4841-969e-925037922a63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:53.584608 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.584545 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26" pod "ebbf75c2-489e-4841-969e-925037922a63" (UID: "ebbf75c2-489e-4841-969e-925037922a63"). InnerVolumeSpecName "kube-api-access-dgg26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:53.584922 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.584896 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs" pod "ebbf75c2-489e-4841-969e-925037922a63" (UID: "ebbf75c2-489e-4841-969e-925037922a63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:53.704579 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.704344 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\") pod \"ebbf75c2-489e-4841-969e-925037922a63\" (UID: \"ebbf75c2-489e-4841-969e-925037922a63\") " Nov 03 01:45:53.705079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.705051 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:53.705079 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.705080 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebbf75c2-489e-4841-969e-925037922a63-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:53.705259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.705094 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgg26\" (UniqueName: \"kubernetes.io/projected/ebbf75c2-489e-4841-969e-925037922a63-kube-api-access-dgg26\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:53.705969 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.705943 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca" pod "ebbf75c2-489e-4841-969e-925037922a63" (UID: "ebbf75c2-489e-4841-969e-925037922a63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:53.806538 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:53.806444 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf75c2-489e-4841-969e-925037922a63-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:45:54.214427 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.214376 4992 generic.go:356] "Generic (PLEG): container finished" podID="ebbf75c2-489e-4841-969e-925037922a63" containerID="e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8" exitCode=0 Nov 03 01:45:54.214966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.214436 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" Nov 03 01:45:54.214966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.214661 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" event={"ID":"ebbf75c2-489e-4841-969e-925037922a63","Type":"ContainerDied","Data":"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8"} Nov 03 01:45:54.214966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.214710 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn" event={"ID":"ebbf75c2-489e-4841-969e-925037922a63","Type":"ContainerDied","Data":"dec73bf4a532cc9a95be511a9421c94b7ae264a33ce5e4dd0f226ad1ca81f29b"} Nov 03 01:45:54.214966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.214742 4992 scope.go:122] "RemoveContainer" containerID="e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8" Nov 03 01:45:54.283090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.283060 4992 scope.go:122] "RemoveContainer" containerID="e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8" Nov 03 01:45:54.283464 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:45:54.283441 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8\": container with ID starting with e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8 not found: ID does not exist" containerID="e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8" Nov 03 01:45:54.283503 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.283478 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8"} err="failed to get container status \"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8\": rpc error: code = NotFound desc = could not find container \"e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8\": container with ID starting with e8d8af54e92cfb2aaa0dba08298ec53c0b81046b10bea3f0d9b3bc8cbeb498f8 not found: ID does not exist" Nov 03 01:45:54.391727 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.391689 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:45:54.397361 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.397325 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-012d31-75d649586f-t4fmn"] Nov 03 01:45:54.596263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:54.596184 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbf75c2-489e-4841-969e-925037922a63" path="/var/lib/kubelet/pods/ebbf75c2-489e-4841-969e-925037922a63/volumes" Nov 03 01:45:56.672786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:56.672747 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:45:56.673259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:56.672964 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" containerID="cri-o://eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671" gracePeriod=30 Nov 03 01:45:57.158853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.158764 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:45:57.592584 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.592537 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:45:57.593184 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.593166 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ebbf75c2-489e-4841-969e-925037922a63" containerName="maas-api" Nov 03 01:45:57.593184 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.593182 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbf75c2-489e-4841-969e-925037922a63" containerName="maas-api" Nov 03 01:45:57.593359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.593275 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbf75c2-489e-4841-969e-925037922a63" containerName="maas-api" Nov 03 01:45:57.598912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.598880 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.660010 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.659966 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:45:57.666467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666433 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.666728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666523 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.666728 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666673 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.666941 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666759 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.666941 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666831 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.667055 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.666963 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.667195 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.667167 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.667252 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.667236 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.667336 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.667321 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcn4x\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.768883 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.768835 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.768916 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.768969 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769039 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcn4x\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769120 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769199 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769240 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769294 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769420 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769504 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769575 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769661 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.769813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769715 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.770073 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.769976 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.772338 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.772309 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.772651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.772627 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.846026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.845937 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcn4x\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.848737 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.848705 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:57.918934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:57.918887 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:58.310427 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.309307 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:45:58.310427 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.309600 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" containerID="cri-o://6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26" gracePeriod=30 Nov 03 01:45:58.316780 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.316069 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:45:58.316780 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.316316 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" containerID="cri-o://0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6" gracePeriod=30 Nov 03 01:45:58.445522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.440438 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:45:58.445522 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:45:58.441380 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7fcfd6_4c4a_425d_bbc6_940c528033ca.slice/crio-64cd9f0ff0d1907127818cf0a85fd0d1f8bfcd98f003f2a0fdd20b2d8eac478f WatchSource:0}: Error finding container 64cd9f0ff0d1907127818cf0a85fd0d1f8bfcd98f003f2a0fdd20b2d8eac478f: Status 404 returned error can't find the container with id 64cd9f0ff0d1907127818cf0a85fd0d1f8bfcd98f003f2a0fdd20b2d8eac478f Nov 03 01:45:58.449325 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.449273 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:58.453619 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.449579 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:58.453619 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:58.452711 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:45:59.305372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.305332 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" event={"ID":"7d7fcfd6-4c4a-425d-bbc6-940c528033ca","Type":"ContainerStarted","Data":"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f"} Nov 03 01:45:59.305900 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.305397 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" event={"ID":"7d7fcfd6-4c4a-425d-bbc6-940c528033ca","Type":"ContainerStarted","Data":"64cd9f0ff0d1907127818cf0a85fd0d1f8bfcd98f003f2a0fdd20b2d8eac478f"} Nov 03 01:45:59.431084 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.431032 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podStartSLOduration=2.431018429 podStartE2EDuration="2.431018429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:59.429273406 +0000 UTC m=+1695.008095993" watchObservedRunningTime="2026-11-03 01:45:59.431018429 +0000 UTC m=+1695.009841016" Nov 03 01:45:59.920071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.920037 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:45:59.921975 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.921949 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:45:59.922092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:45:59.922003 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 01:46:00.920335 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:00.920300 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:46:00.920762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:00.920357 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 01:46:01.920593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:01.919922 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:46:01.920593 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:01.919980 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 01:46:02.010686 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.010659 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:46:02.097617 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097547 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.097822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097649 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.097822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097677 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.097822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097726 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.097822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097771 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097874 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097909 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097931 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097951 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097981 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.098057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.097993 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:02.098368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098142 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\") pod \"7f98310c-805b-4d63-aa97-cff472ad4856\" (UID: \"7f98310c-805b-4d63-aa97-cff472ad4856\") " Nov 03 01:46:02.098368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098150 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:02.098515 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098485 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:02.098934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098905 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.099026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098937 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98310c-805b-4d63-aa97-cff472ad4856-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.099026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098953 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.099026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098967 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.099026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.098982 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.100353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.100321 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:02.100468 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.100357 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "kube-api-access-msc6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:02.100534 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.100505 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:02.100627 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.100576 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo" pod "7f98310c-805b-4d63-aa97-cff472ad4856" (UID: "7f98310c-805b-4d63-aa97-cff472ad4856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:02.152408 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:02.152365 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:46:02.200938 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.200903 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98310c-805b-4d63-aa97-cff472ad4856-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.200938 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.200936 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.201164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.200949 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msc6z\" (UniqueName: \"kubernetes.io/projected/7f98310c-805b-4d63-aa97-cff472ad4856-kube-api-access-msc6z\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.201164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.200960 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98310c-805b-4d63-aa97-cff472ad4856-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:02.326689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.326599 4992 generic.go:356] "Generic (PLEG): container finished" podID="7f98310c-805b-4d63-aa97-cff472ad4856" containerID="eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671" exitCode=0 Nov 03 01:46:02.326689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.326652 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" Nov 03 01:46:02.326871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.326685 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" event={"ID":"7f98310c-805b-4d63-aa97-cff472ad4856","Type":"ContainerDied","Data":"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671"} Nov 03 01:46:02.326871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.326761 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" event={"ID":"7f98310c-805b-4d63-aa97-cff472ad4856","Type":"ContainerDied","Data":"fcc71227d33ff1e6c17aca5bc564753f817209f52d45c6882caf281befa8f66a"} Nov 03 01:46:02.326871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.326784 4992 scope.go:122] "RemoveContainer" containerID="eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671" Nov 03 01:46:02.338183 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.338163 4992 scope.go:122] "RemoveContainer" containerID="eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671" Nov 03 01:46:02.338455 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:02.338435 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671\": container with ID starting with eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671 not found: ID does not exist" containerID="eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671" Nov 03 01:46:02.338504 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.338466 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671"} err="failed to get container status \"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671\": rpc error: code = NotFound desc = could not find container \"eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671\": container with ID starting with eda212ff9d13b99c27760dabe2870b38c63a32bd1a932b2b451ca729a1a4d671 not found: ID does not exist" Nov 03 01:46:02.502355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.502314 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:46:02.506862 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.506833 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc"] Nov 03 01:46:02.596938 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.596857 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" path="/var/lib/kubelet/pods/7f98310c-805b-4d63-aa97-cff472ad4856/volumes" Nov 03 01:46:02.920074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.920038 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:46:02.920263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.920108 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 01:46:02.946671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.946625 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:46:02.947074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:02.946697 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-vzndc" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:46:03.660387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.660354 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:46:03.719878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.719850 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:46:03.826401 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826355 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.826401 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826422 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826444 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826482 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826507 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826528 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826590 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826641 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826700 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.826742 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826743 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826761 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826778 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826770 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826882 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826924 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826956 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.826972 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827054 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.827143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827129 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827140 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827155 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827288 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\") pod \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\" (UID: \"e26fe2a8-8291-46a8-aca6-ffcc8a4229af\") " Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827344 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827347 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\") pod \"52ba4342-0784-43f1-8840-b7b4400f5c04\" (UID: \"52ba4342-0784-43f1-8840-b7b4400f5c04\") " Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827392 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:03.827642 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827551 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.828057 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.827814 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828192 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828221 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828237 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828255 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828269 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828283 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ba4342-0784-43f1-8840-b7b4400f5c04-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828296 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828312 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.828637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.828327 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.829656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829436 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.829656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829537 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "kube-api-access-49gp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.829656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829553 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:03.829905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829746 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.829905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829779 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.829905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829806 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:03.830022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.829983 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.830108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.830088 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh" pod "e26fe2a8-8291-46a8-aca6-ffcc8a4229af" (UID: "e26fe2a8-8291-46a8-aca6-ffcc8a4229af"). InnerVolumeSpecName "kube-api-access-766bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:03.830831 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.830812 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo" pod "52ba4342-0784-43f1-8840-b7b4400f5c04" (UID: "52ba4342-0784-43f1-8840-b7b4400f5c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:03.919661 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.919622 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 01:46:03.919880 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.919691 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 01:46:03.929982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.929948 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766bh\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-kube-api-access-766bh\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.929982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.929980 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.929995 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930008 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gp6\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-kube-api-access-49gp6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930021 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930033 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930047 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930060 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ba4342-0784-43f1-8840-b7b4400f5c04-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:03.930182 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:03.930072 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e26fe2a8-8291-46a8-aca6-ffcc8a4229af-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:04.145846 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.145813 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:46:04.146343 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.146027 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" containerID="cri-o://f331c712d24220ad8ede2ac91bc6f581fe9f59ffe651230737712de91b237c9b" gracePeriod=30 Nov 03 01:46:04.343844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.343754 4992 generic.go:356] "Generic (PLEG): container finished" podID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerID="6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26" exitCode=0 Nov 03 01:46:04.343844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.343802 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" Nov 03 01:46:04.344039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.343846 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" event={"ID":"e26fe2a8-8291-46a8-aca6-ffcc8a4229af","Type":"ContainerDied","Data":"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26"} Nov 03 01:46:04.344039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.343912 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv" event={"ID":"e26fe2a8-8291-46a8-aca6-ffcc8a4229af","Type":"ContainerDied","Data":"d07215227a485cd6a199398ade5dc4e7108fd7a8e6fe4bda634e4c3a7616e752"} Nov 03 01:46:04.344039 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.343942 4992 scope.go:122] "RemoveContainer" containerID="6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26" Nov 03 01:46:04.347078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.347054 4992 generic.go:356] "Generic (PLEG): container finished" podID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerID="0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6" exitCode=0 Nov 03 01:46:04.347199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.347110 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" event={"ID":"52ba4342-0784-43f1-8840-b7b4400f5c04","Type":"ContainerDied","Data":"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6"} Nov 03 01:46:04.347199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.347129 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" Nov 03 01:46:04.347199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.347145 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w" event={"ID":"52ba4342-0784-43f1-8840-b7b4400f5c04","Type":"ContainerDied","Data":"733170f4d22c46785740e5e18caef6b6630a924ce03c17b65c87493c4533c5bc"} Nov 03 01:46:04.361005 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.360981 4992 scope.go:122] "RemoveContainer" containerID="6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26" Nov 03 01:46:04.361344 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:04.361322 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26\": container with ID starting with 6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26 not found: ID does not exist" containerID="6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26" Nov 03 01:46:04.361422 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.361353 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26"} err="failed to get container status \"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26\": rpc error: code = NotFound desc = could not find container \"6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26\": container with ID starting with 6411a827db2489a882c3135fb887ca71495505027fd6b5c78c67f0d3f3f5bd26 not found: ID does not exist" Nov 03 01:46:04.361422 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.361371 4992 scope.go:122] "RemoveContainer" containerID="0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6" Nov 03 01:46:04.398801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.398770 4992 scope.go:122] "RemoveContainer" containerID="0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6" Nov 03 01:46:04.399211 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:04.399178 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6\": container with ID starting with 0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6 not found: ID does not exist" containerID="0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6" Nov 03 01:46:04.399327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.399223 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6"} err="failed to get container status \"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6\": rpc error: code = NotFound desc = could not find container \"0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6\": container with ID starting with 0cfe9f2459ed3ad4088eaa078393e654ae619d876586cfa8080e45b57ca00cd6 not found: ID does not exist" Nov 03 01:46:04.517994 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.517960 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:46:04.521008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.520980 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-bx9dv"] Nov 03 01:46:04.599052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.598972 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" path="/var/lib/kubelet/pods/e26fe2a8-8291-46a8-aca6-ffcc8a4229af/volumes" Nov 03 01:46:04.636287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.636253 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:46:04.639788 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.639758 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-gvf6w"] Nov 03 01:46:04.923996 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.923964 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:46:04.924473 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.924454 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:46:04.925204 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:04.925187 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:46:06.597574 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:06.597527 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" path="/var/lib/kubelet/pods/52ba4342-0784-43f1-8840-b7b4400f5c04/volumes" Nov 03 01:46:09.386762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.386729 4992 generic.go:356] "Generic (PLEG): container finished" podID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerID="f331c712d24220ad8ede2ac91bc6f581fe9f59ffe651230737712de91b237c9b" exitCode=0 Nov 03 01:46:09.387178 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.386801 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" event={"ID":"3b136c78-c96b-4bfb-a152-bf9834f63459","Type":"ContainerDied","Data":"f331c712d24220ad8ede2ac91bc6f581fe9f59ffe651230737712de91b237c9b"} Nov 03 01:46:09.434885 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.434817 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:46:09.504887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.504836 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.504887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.504887 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505032 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505069 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505108 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505160 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.505313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505174 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505313 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505244 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.505487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505324 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505351 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505418 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:09.505487 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505442 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\") pod \"3b136c78-c96b-4bfb-a152-bf9834f63459\" (UID: \"3b136c78-c96b-4bfb-a152-bf9834f63459\") " Nov 03 01:46:09.505729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505601 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.505827 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.505793 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.506289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.506263 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.506413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.506292 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.506413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.506307 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b136c78-c96b-4bfb-a152-bf9834f63459-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.506413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.506320 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.506413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.506333 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.507541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.507516 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.507541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.507532 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:09.507750 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.507731 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:09.507801 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.507776 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n" pod "3b136c78-c96b-4bfb-a152-bf9834f63459" (UID: "3b136c78-c96b-4bfb-a152-bf9834f63459"). InnerVolumeSpecName "kube-api-access-nhz5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:09.607440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.607385 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhz5n\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-kube-api-access-nhz5n\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.607440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.607424 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.607440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.607440 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:09.607782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:09.607455 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b136c78-c96b-4bfb-a152-bf9834f63459-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:10.398192 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:10.398151 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" event={"ID":"3b136c78-c96b-4bfb-a152-bf9834f63459","Type":"ContainerDied","Data":"ab77999a890edfce1b5b6fcfe0d64a690319bcc9a332b0929298beb0c81549df"} Nov 03 01:46:10.398821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:10.398798 4992 scope.go:122] "RemoveContainer" containerID="f331c712d24220ad8ede2ac91bc6f581fe9f59ffe651230737712de91b237c9b" Nov 03 01:46:10.399151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:10.399131 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg" Nov 03 01:46:10.641307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:10.641275 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:46:10.693224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:10.693190 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-d8pfg"] Nov 03 01:46:12.596588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:12.596538 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" path="/var/lib/kubelet/pods/3b136c78-c96b-4bfb-a152-bf9834f63459/volumes" Nov 03 01:46:15.748594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.748538 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:46:15.749173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749157 4992 cpu_manager.go:414] "containerMap: removing container" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" Nov 03 01:46:15.749221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749174 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" Nov 03 01:46:15.749221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749193 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" Nov 03 01:46:15.749221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749198 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" Nov 03 01:46:15.749221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749216 4992 cpu_manager.go:414] "containerMap: removing container" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" Nov 03 01:46:15.749221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749221 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749239 4992 cpu_manager.go:414] "containerMap: removing container" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749244 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749328 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba4342-0784-43f1-8840-b7b4400f5c04" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749340 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f98310c-805b-4d63-aa97-cff472ad4856" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749348 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26fe2a8-8291-46a8-aca6-ffcc8a4229af" containerName="istio-proxy" Nov 03 01:46:15.749375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.749356 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b136c78-c96b-4bfb-a152-bf9834f63459" containerName="istio-proxy" Nov 03 01:46:15.754438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.754414 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.785826 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.785778 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.785826 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.785825 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.786049 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.785998 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.786089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.786067 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.806790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.806757 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:46:15.808877 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.808848 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2b9f22c1-serving-cert\"" Nov 03 01:46:15.887291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.887255 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.887291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.887294 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.887496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.887389 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.887496 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.887451 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.887649 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:15.887598 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-serving-cert: secret "maas-api-e2e-mt-2b9f22c1-serving-cert" not found Nov 03 01:46:15.887702 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:15.887687 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls podName:93e2fa35-10cf-41ce-8675-7e6b0727f5c5 nodeName:}" failed. No retries permitted until 2026-11-03 01:46:16.387666192 +0000 UTC m=+1711.966488773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls") pod "maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" (UID: "93e2fa35-10cf-41ce-8675-7e6b0727f5c5") : secret "maas-api-e2e-mt-2b9f22c1-serving-cert" not found Nov 03 01:46:15.887940 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.887918 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.890052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.890026 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:15.950809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:15.950778 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:16.394670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:16.394633 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:16.397128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:16.397100 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") pod \"maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:16.668550 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:16.668515 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:17.047207 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:46:17.047170 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e2fa35_10cf_41ce_8675_7e6b0727f5c5.slice/crio-b29dc739a1355501ca1d706c2521d0857006d30dfc582b8536f333469af21e94 WatchSource:0}: Error finding container b29dc739a1355501ca1d706c2521d0857006d30dfc582b8536f333469af21e94: Status 404 returned error can't find the container with id b29dc739a1355501ca1d706c2521d0857006d30dfc582b8536f333469af21e94 Nov 03 01:46:17.048715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:17.048695 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:46:17.050138 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:17.050086 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:46:17.448800 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:17.448762 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" event={"ID":"93e2fa35-10cf-41ce-8675-7e6b0727f5c5","Type":"ContainerStarted","Data":"b29dc739a1355501ca1d706c2521d0857006d30dfc582b8536f333469af21e94"} Nov 03 01:46:18.457291 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:18.457250 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" event={"ID":"93e2fa35-10cf-41ce-8675-7e6b0727f5c5","Type":"ContainerStarted","Data":"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd"} Nov 03 01:46:18.457803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:18.457314 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:18.581384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:18.581333 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" podStartSLOduration=2.68033469 podStartE2EDuration="3.581318414s" totalImagesPullingTime="900.983724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:18.578845807 +0000 UTC m=+1714.157668398" watchObservedRunningTime="2026-11-03 01:46:18.581318414 +0000 UTC m=+1714.160141001" Nov 03 01:46:24.470261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:24.470229 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:46:32.153058 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:32.153014 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:46:35.637189 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:35.637149 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:46:35.637806 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:35.637441 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" containerID="cri-o://1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6" gracePeriod=30 Nov 03 01:46:41.355735 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.355664 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:46:41.360966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.360939 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.360991 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361065 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361205 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361315 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361290 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361369 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361331 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361422 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361369 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361421 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.361527 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.361491 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\") pod \"660224da-d543-4bea-85e0-c7afc039a780\" (UID: \"660224da-d543-4bea-85e0-c7afc039a780\") " Nov 03 01:46:41.393583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.386586 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:41.393583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.387463 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.409589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.404678 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:41.420794 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.420739 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:41.425603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.421944 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:41.432596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.427659 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:41.441585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.436788 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "kube-api-access-nmckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:41.441585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.436889 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:46:41.441585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.436955 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:41.441585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.437005 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy" pod "660224da-d543-4bea-85e0-c7afc039a780" (UID: "660224da-d543-4bea-85e0-c7afc039a780"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:46:41.493887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.493845 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494125 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494253 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660224da-d543-4bea-85e0-c7afc039a780-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494371 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494357 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494468 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494454 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494574 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494544 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660224da-d543-4bea-85e0-c7afc039a780-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494672 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660224da-d543-4bea-85e0-c7afc039a780-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.494786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.494769 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmckq\" (UniqueName: \"kubernetes.io/projected/660224da-d543-4bea-85e0-c7afc039a780-kube-api-access-nmckq\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:46:41.742582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.741713 4992 generic.go:356] "Generic (PLEG): container finished" podID="660224da-d543-4bea-85e0-c7afc039a780" containerID="1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6" exitCode=0 Nov 03 01:46:41.742582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.741780 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" event={"ID":"660224da-d543-4bea-85e0-c7afc039a780","Type":"ContainerDied","Data":"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6"} Nov 03 01:46:41.742582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.741839 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" event={"ID":"660224da-d543-4bea-85e0-c7afc039a780","Type":"ContainerDied","Data":"284eda1b0b4957c4a39147a79b44de834115a9279774a426e34a5a48f594d495"} Nov 03 01:46:41.742582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.741872 4992 scope.go:122] "RemoveContainer" containerID="1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6" Nov 03 01:46:41.742582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.742068 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx" Nov 03 01:46:41.761539 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.761450 4992 scope.go:122] "RemoveContainer" containerID="1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6" Nov 03 01:46:41.790866 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:46:41.790806 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6\": container with ID starting with 1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6 not found: ID does not exist" containerID="1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6" Nov 03 01:46:41.791193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.791138 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6"} err="failed to get container status \"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6\": rpc error: code = NotFound desc = could not find container \"1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6\": container with ID starting with 1d07d3c12251ba6ac69883e5b956e19d02cc2f89c7bea046dcf1b2896f3ba1f6 not found: ID does not exist" Nov 03 01:46:41.952008 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.951971 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:46:41.958164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:41.958126 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-h76sx"] Nov 03 01:46:42.595838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:42.595805 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660224da-d543-4bea-85e0-c7afc039a780" path="/var/lib/kubelet/pods/660224da-d543-4bea-85e0-c7afc039a780/volumes" Nov 03 01:46:58.867934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.867851 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:46:58.868591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.868543 4992 cpu_manager.go:414] "containerMap: removing container" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" Nov 03 01:46:58.868591 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.868575 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" Nov 03 01:46:58.868790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.868728 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="660224da-d543-4bea-85e0-c7afc039a780" containerName="istio-proxy" Nov 03 01:46:58.873839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.873818 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.894936 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.894902 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.894949 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.894969 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895203 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895163 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895275 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895330 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895308 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895402 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895345 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8xn\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895468 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895449 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.895505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.895486 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.928130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.928100 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b9f22c1-openshift-default-dockercfg-p5xl7\"" Nov 03 01:46:58.983312 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.983279 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:46:58.996779 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.996734 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.996982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.996864 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.996982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.996900 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.996982 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.996956 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8xn\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997032 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997073 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997110 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997150 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997176 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997299 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997231 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997573 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997537 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.997851 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997823 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.998018 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.997999 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:58.998341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:58.998318 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.000471 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.000443 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.000934 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.000912 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.060118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.060080 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.060118 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.060112 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8xn\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.187099 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.187060 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:46:59.660693 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:46:59.660657 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420e913d_1162_402f_9234_90c59e4f87cb.slice/crio-8bcc4d8c9e58a48c7e23d2ffd6fd815a75b37c4bf6d5b4e32c773026edbcf2ca WatchSource:0}: Error finding container 8bcc4d8c9e58a48c7e23d2ffd6fd815a75b37c4bf6d5b4e32c773026edbcf2ca: Status 404 returned error can't find the container with id 8bcc4d8c9e58a48c7e23d2ffd6fd815a75b37c4bf6d5b4e32c773026edbcf2ca Nov 03 01:46:59.663245 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.663218 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:46:59.663337 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.663301 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:46:59.663395 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.663379 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:46:59.663431 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.663417 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:46:59.869981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.869935 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" event={"ID":"420e913d-1162-402f-9234-90c59e4f87cb","Type":"ContainerStarted","Data":"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab"} Nov 03 01:46:59.870375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.870029 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" event={"ID":"420e913d-1162-402f-9234-90c59e4f87cb","Type":"ContainerStarted","Data":"8bcc4d8c9e58a48c7e23d2ffd6fd815a75b37c4bf6d5b4e32c773026edbcf2ca"} Nov 03 01:46:59.994637 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:46:59.994579 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podStartSLOduration=1.994547528 podStartE2EDuration="1.994547528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:59.993018907 +0000 UTC m=+1755.571841493" watchObservedRunningTime="2026-11-03 01:46:59.994547528 +0000 UTC m=+1755.573370116" Nov 03 01:47:00.187603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:00.187546 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:47:00.189703 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:00.189668 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:00.189874 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:00.189746 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:01.188040 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:01.187998 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:01.188616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:01.188074 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:02.152572 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:47:02.152519 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:47:02.188064 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:02.188026 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:02.188478 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:02.188079 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:03.187708 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:03.187669 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:03.187919 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:03.187733 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:04.188002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:04.187964 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:04.188410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:04.188022 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:05.188580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:05.188009 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:05.188580 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:05.188075 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:06.188132 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:06.188085 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 01:47:06.188287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:06.188154 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 01:47:07.193152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:07.193121 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:47:07.193630 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:07.193591 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:47:07.194352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:07.194333 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:47:14.006046 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.006005 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:47:14.010647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.010623 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128335 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128295 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128350 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128473 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqspp\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128530 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128662 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128766 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128816 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.128979 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.128960 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.129040 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.129004 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.227270 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.227223 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:47:14.230376 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.230320 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqspp\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.230640 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.230616 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.230849 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.230831 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231013 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231233 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231213 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231437 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231417 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231614 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231594 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231745 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.231880 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231865 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.232000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231970 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.232068 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231985 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.232068 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.231666 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.232238 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.232217 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.232505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.232486 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.244358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.244119 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.244358 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.244321 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.293155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.293077 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqspp\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.296703 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.296669 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.334921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.334874 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:14.867598 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:47:14.867545 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055c1fe3_3f0d_46d0_b238_7f007eed1591.slice/crio-ac43a6cb63d47bb5729347de3ea06928d1ee6ed4025694950b07786d485d77a1 WatchSource:0}: Error finding container ac43a6cb63d47bb5729347de3ea06928d1ee6ed4025694950b07786d485d77a1: Status 404 returned error can't find the container with id ac43a6cb63d47bb5729347de3ea06928d1ee6ed4025694950b07786d485d77a1 Nov 03 01:47:14.870451 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.870413 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:14.870575 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.870500 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:14.870649 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.870546 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:14.870838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.870813 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:47:14.986388 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.986342 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" event={"ID":"055c1fe3-3f0d-46d0-b238-7f007eed1591","Type":"ContainerStarted","Data":"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02"} Nov 03 01:47:14.986603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:14.986422 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" event={"ID":"055c1fe3-3f0d-46d0-b238-7f007eed1591","Type":"ContainerStarted","Data":"ac43a6cb63d47bb5729347de3ea06928d1ee6ed4025694950b07786d485d77a1"} Nov 03 01:47:15.112981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:15.112926 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podStartSLOduration=2.112911442 podStartE2EDuration="2.112911442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:15.109542538 +0000 UTC m=+1770.688365124" watchObservedRunningTime="2026-11-03 01:47:15.112911442 +0000 UTC m=+1770.691734025" Nov 03 01:47:15.337352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:15.337313 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:15.338549 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:15.338519 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:15.338682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:15.338586 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:16.337115 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:16.337074 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:16.337659 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:16.337147 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:17.336871 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:17.336831 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:17.337068 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:17.336892 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:18.337575 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:18.337528 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:18.338020 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:18.337621 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:19.337397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:19.337358 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:19.337639 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:19.337414 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:20.341121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:20.340069 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 01:47:20.341121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:20.340126 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 01:47:21.525538 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:21.525501 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:21.526804 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:21.526776 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:21.528111 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:21.528087 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:47:27.730430 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:27.730389 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:47:27.730888 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:27.730613 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" podUID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" containerName="maas-api" containerID="cri-o://a0bfe00829853eabd552c5b3be898956aea09e4f190d7b6d99df04cf4118e2ce" gracePeriod=30 Nov 03 01:47:28.084155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.080374 4992 generic.go:356] "Generic (PLEG): container finished" podID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" containerID="a0bfe00829853eabd552c5b3be898956aea09e4f190d7b6d99df04cf4118e2ce" exitCode=0 Nov 03 01:47:28.084155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.080415 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" event={"ID":"d3e31f55-bd9e-4d32-8283-2fe1e13ba482","Type":"ContainerDied","Data":"a0bfe00829853eabd552c5b3be898956aea09e4f190d7b6d99df04cf4118e2ce"} Nov 03 01:47:28.147776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.147742 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:47:28.259137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.259103 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\") pod \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " Nov 03 01:47:28.259371 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.259355 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\") pod \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " Nov 03 01:47:28.259470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.259459 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\" (UniqueName: \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\") pod \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " Nov 03 01:47:28.259616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.259604 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") pod \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\" (UID: \"d3e31f55-bd9e-4d32-8283-2fe1e13ba482\") " Nov 03 01:47:28.260241 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.260211 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca" pod "d3e31f55-bd9e-4d32-8283-2fe1e13ba482" (UID: "d3e31f55-bd9e-4d32-8283-2fe1e13ba482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:28.260490 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.260470 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.262312 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.262286 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t" pod "d3e31f55-bd9e-4d32-8283-2fe1e13ba482" (UID: "d3e31f55-bd9e-4d32-8283-2fe1e13ba482"). InnerVolumeSpecName "kube-api-access-zt28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:28.262465 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.262444 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs" pod "d3e31f55-bd9e-4d32-8283-2fe1e13ba482" (UID: "d3e31f55-bd9e-4d32-8283-2fe1e13ba482"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:28.262513 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.262500 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls" pod "d3e31f55-bd9e-4d32-8283-2fe1e13ba482" (UID: "d3e31f55-bd9e-4d32-8283-2fe1e13ba482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:28.361628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.361516 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.361628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.361548 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt28t\" (UniqueName: \"kubernetes.io/projected/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-kube-api-access-zt28t\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:28.361628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:28.361581 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3e31f55-bd9e-4d32-8283-2fe1e13ba482-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:29.088671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.088629 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" event={"ID":"d3e31f55-bd9e-4d32-8283-2fe1e13ba482","Type":"ContainerDied","Data":"8a853f8c6b6fe3a84ed7c2ebbd6601f240756f3c151f493b7ec12ae5a4f7370a"} Nov 03 01:47:29.088671 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.088673 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9" Nov 03 01:47:29.089155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.088705 4992 scope.go:122] "RemoveContainer" containerID="a0bfe00829853eabd552c5b3be898956aea09e4f190d7b6d99df04cf4118e2ce" Nov 03 01:47:29.116616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.116582 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:47:29.117193 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.117180 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" containerName="maas-api" Nov 03 01:47:29.117239 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.117195 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" containerName="maas-api" Nov 03 01:47:29.117321 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.117311 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" containerName="maas-api" Nov 03 01:47:29.122283 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.122260 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184421 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184386 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184615 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184700 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184659 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184806 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184849 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.184990 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.184975 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.185054 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.185006 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.185054 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.185040 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.185162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.185105 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkbg\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.189467 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.189437 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:47:29.287043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287001 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287067 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287102 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287186 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287209 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287240 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287374 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkbg\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287456 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287761 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287725 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287757 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287795 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.287853 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287840 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.288012 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.287953 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.288263 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.288242 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.290163 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.290141 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.290426 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.290406 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.350674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.350601 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.350915 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.350889 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkbg\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.386508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.386473 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:47:29.390527 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.390501 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4952b1-779c849d69-l26p9"] Nov 03 01:47:29.436089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.436049 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:29.865984 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:47:29.865950 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ff482_4a35_4e2d_b6a2_6005c07d128c.slice/crio-47c8ad2d5f38e3f94ba495e4cc2a5f2d2169ba91c59f7fd02666a06e5ef31619 WatchSource:0}: Error finding container 47c8ad2d5f38e3f94ba495e4cc2a5f2d2169ba91c59f7fd02666a06e5ef31619: Status 404 returned error can't find the container with id 47c8ad2d5f38e3f94ba495e4cc2a5f2d2169ba91c59f7fd02666a06e5ef31619 Nov 03 01:47:29.868922 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.868897 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:47:29.869307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.869270 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:29.869372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.869356 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:29.869414 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:29.869400 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:47:30.097351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.097309 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" event={"ID":"1c8ff482-4a35-4e2d-b6a2-6005c07d128c","Type":"ContainerStarted","Data":"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816"} Nov 03 01:47:30.097912 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.097386 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" event={"ID":"1c8ff482-4a35-4e2d-b6a2-6005c07d128c","Type":"ContainerStarted","Data":"47c8ad2d5f38e3f94ba495e4cc2a5f2d2169ba91c59f7fd02666a06e5ef31619"} Nov 03 01:47:30.223121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.223057 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podStartSLOduration=1.223035724 podStartE2EDuration="1.223035724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:30.221476931 +0000 UTC m=+1785.800299541" watchObservedRunningTime="2026-11-03 01:47:30.223035724 +0000 UTC m=+1785.801858309" Nov 03 01:47:30.436394 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.436353 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:30.438319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.438290 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:30.438438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.438346 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:30.596412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:30.596322 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e31f55-bd9e-4d32-8283-2fe1e13ba482" path="/var/lib/kubelet/pods/d3e31f55-bd9e-4d32-8283-2fe1e13ba482/volumes" Nov 03 01:47:31.436570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:31.436527 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:31.436999 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:31.436637 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:32.152114 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:47:32.152063 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:47:32.436814 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:32.436773 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:32.437232 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:32.436860 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:33.436770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:33.436721 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:33.436955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:33.436787 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:34.458345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:34.458307 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:34.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:34.459030 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:35.436535 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:35.436494 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:35.436763 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:35.436552 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:36.007281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:36.007230 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:47:36.008570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:36.008511 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" containerID="cri-o://5525e2a4f414bc3b32f25f66c01cb005bf826c9c424e5c3c37d857739f3b8a95" gracePeriod=30 Nov 03 01:47:36.436743 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:36.436696 4992 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 01:47:36.436901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:36.436772 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 01:47:37.442145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:37.442113 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:37.442615 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:37.442591 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:37.443482 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:37.443460 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:47:37.829807 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:37.829757 4992 patch_prober.go:29] interesting pod/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:47:37.830090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:37.830050 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:47:39.083019 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:39.082974 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:47:39.083893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:39.083854 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" containerID="cri-o://6185f9bdee57d2567c58312a64e7ea8c55ff3eb1a32a7f865832d55d7f75a7ef" gracePeriod=30 Nov 03 01:47:41.264214 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.264176 4992 generic.go:356] "Generic (PLEG): container finished" podID="b442232f-a4b4-459a-b851-63203d3d9bce" containerID="5525e2a4f414bc3b32f25f66c01cb005bf826c9c424e5c3c37d857739f3b8a95" exitCode=0 Nov 03 01:47:41.264696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.264249 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerDied","Data":"5525e2a4f414bc3b32f25f66c01cb005bf826c9c424e5c3c37d857739f3b8a95"} Nov 03 01:47:41.264696 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.264333 4992 scope.go:122] "RemoveContainer" containerID="7de84c6a5fb82fa0e7c6d65a22ba99cf24c9d6a0cbcab74b8596c13189cd36b3" Nov 03 01:47:41.292421 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.292399 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:47:41.372894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.372861 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.372967 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373037 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373102 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373122 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373157 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373202 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373287 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373258 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373324 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373407 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:41.373725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373451 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\") pod \"b442232f-a4b4-459a-b851-63203d3d9bce\" (UID: \"b442232f-a4b4-459a-b851-63203d3d9bce\") " Nov 03 01:47:41.373725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373599 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:41.373993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373709 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:41.373993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.373901 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:41.374160 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.374138 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442232f-a4b4-459a-b851-63203d3d9bce-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.374221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.374168 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.374221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.374184 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.374221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.374198 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.374221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.374212 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.375448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.375422 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:41.375448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.375442 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "kube-api-access-j8pjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:41.375618 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.375460 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:41.375689 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.375674 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo" pod "b442232f-a4b4-459a-b851-63203d3d9bce" (UID: "b442232f-a4b4-459a-b851-63203d3d9bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:47:41.475813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.475770 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442232f-a4b4-459a-b851-63203d3d9bce-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.475813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.475811 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.476060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.475825 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442232f-a4b4-459a-b851-63203d3d9bce-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:41.476060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:41.475839 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pjx\" (UniqueName: \"kubernetes.io/projected/b442232f-a4b4-459a-b851-63203d3d9bce-kube-api-access-j8pjx\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:42.272482 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.272453 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" Nov 03 01:47:42.272921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.272506 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c" event={"ID":"b442232f-a4b4-459a-b851-63203d3d9bce","Type":"ContainerDied","Data":"1ae966d70e824a1ac1fd8f9a2869e242808712c2e6712f8608d2010d4ef82656"} Nov 03 01:47:42.272921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.272582 4992 scope.go:122] "RemoveContainer" containerID="5525e2a4f414bc3b32f25f66c01cb005bf826c9c424e5c3c37d857739f3b8a95" Nov 03 01:47:42.567413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.567326 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:47:42.577027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.572842 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4952b1-openshift-default-5d476677d9-s988c"] Nov 03 01:47:42.595048 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:42.595009 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" path="/var/lib/kubelet/pods/b442232f-a4b4-459a-b851-63203d3d9bce/volumes" Nov 03 01:47:43.721484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:43.721447 4992 patch_prober.go:29] interesting pod/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:47:43.721950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:43.721512 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:47:44.288822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.288787 4992 generic.go:356] "Generic (PLEG): container finished" podID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerID="6185f9bdee57d2567c58312a64e7ea8c55ff3eb1a32a7f865832d55d7f75a7ef" exitCode=0 Nov 03 01:47:44.288995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.288855 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerDied","Data":"6185f9bdee57d2567c58312a64e7ea8c55ff3eb1a32a7f865832d55d7f75a7ef"} Nov 03 01:47:44.288995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.288922 4992 scope.go:122] "RemoveContainer" containerID="a4fd4987984af279735c2f58f290f262c75ab40ee9bcfb65a5b72f29920f978c" Nov 03 01:47:44.670863 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:47:44.669887 4992 info.go:109] Failed to get network devices: open /sys/class/net/3a333d6c46aa9ff/address: no such file or directory Nov 03 01:47:44.768002 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.767976 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:47:44.795269 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.795236 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:47:44.798692 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.798664 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:47:44.820501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820471 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820655 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820513 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820655 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820584 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820655 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820640 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820701 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820733 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820802 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:44.820943 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820849 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.820997 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.820960 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:44.821058 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821016 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.821113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821072 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\") pod \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\" (UID: \"b336bf7a-8c4a-45e9-b7a4-93505278f1a4\") " Nov 03 01:47:44.821113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821080 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:44.821191 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821114 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:44.821436 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821401 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:44.821734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821708 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.821809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821740 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.821809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821757 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.821809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821771 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.821809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.821785 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.823086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.823067 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:47:44.823240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.823222 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:47:44.823393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.823373 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:44.823436 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.823399 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj" pod "b336bf7a-8c4a-45e9-b7a4-93505278f1a4" (UID: "b336bf7a-8c4a-45e9-b7a4-93505278f1a4"). InnerVolumeSpecName "kube-api-access-khqjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:44.923669 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.923629 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.923669 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.923670 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.923939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.923684 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:44.923939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:44.923694 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqjj\" (UniqueName: \"kubernetes.io/projected/b336bf7a-8c4a-45e9-b7a4-93505278f1a4-kube-api-access-khqjj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:47:45.163822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.163782 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:47:45.298045 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.297967 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" Nov 03 01:47:45.298045 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.297984 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8" event={"ID":"b336bf7a-8c4a-45e9-b7a4-93505278f1a4","Type":"ContainerDied","Data":"3a333d6c46aa9ff5d7d17e6b20926427bd06520a4a3a7d85d3adf57db3403f7e"} Nov 03 01:47:45.298229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.298048 4992 scope.go:122] "RemoveContainer" containerID="6185f9bdee57d2567c58312a64e7ea8c55ff3eb1a32a7f865832d55d7f75a7ef" Nov 03 01:47:45.470296 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.470260 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:47:45.475999 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:45.475966 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5e048b-openshift-default-6955d56565-sbft8"] Nov 03 01:47:46.597453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:47:46.597416 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" path="/var/lib/kubelet/pods/b336bf7a-8c4a-45e9-b7a4-93505278f1a4/volumes" Nov 03 01:48:00.185527 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.185462 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerName="main" containerID="cri-o://966f695bd150174faf4bc2a425076de6dcd68335ebc963d3c720e287b13b0ccd" gracePeriod=45 Nov 03 01:48:00.359597 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.359008 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Nov 03 01:48:00.421899 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.421864 4992 generic.go:356] "Generic (PLEG): container finished" podID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerID="966f695bd150174faf4bc2a425076de6dcd68335ebc963d3c720e287b13b0ccd" exitCode=0 Nov 03 01:48:00.422110 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.421934 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" event={"ID":"cf462ddc-c83e-4aa3-8162-9b609cef4528","Type":"ContainerDied","Data":"966f695bd150174faf4bc2a425076de6dcd68335ebc963d3c720e287b13b0ccd"} Nov 03 01:48:00.519878 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.519848 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:48:00.576453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576411 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.576672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576623 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.576746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576675 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.576746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576731 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.576898 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576863 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:00.576962 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.576885 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.577049 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577028 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\" (UniqueName: \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\") pod \"cf462ddc-c83e-4aa3-8162-9b609cef4528\" (UID: \"cf462ddc-c83e-4aa3-8162-9b609cef4528\") " Nov 03 01:48:00.577049 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577036 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:00.577252 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577142 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:00.577611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577550 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:00.577688 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577618 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:00.577688 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.577631 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:00.579119 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.579091 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:00.579224 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.579146 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:00.579295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.579277 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz" pod "cf462ddc-c83e-4aa3-8162-9b609cef4528" (UID: "cf462ddc-c83e-4aa3-8162-9b609cef4528"). InnerVolumeSpecName "kube-api-access-2h4wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:00.681797 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.681761 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf462ddc-c83e-4aa3-8162-9b609cef4528-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:00.681797 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.681796 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf462ddc-c83e-4aa3-8162-9b609cef4528-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:00.682047 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:00.681814 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4wz\" (UniqueName: \"kubernetes.io/projected/cf462ddc-c83e-4aa3-8162-9b609cef4528-kube-api-access-2h4wz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:01.459581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:01.455171 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" event={"ID":"cf462ddc-c83e-4aa3-8162-9b609cef4528","Type":"ContainerDied","Data":"f451fe6c1d97a6f1bdae642f3f248e2d35d6279f9b2508717537f4ec89df9d59"} Nov 03 01:48:01.459581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:01.455248 4992 scope.go:122] "RemoveContainer" containerID="966f695bd150174faf4bc2a425076de6dcd68335ebc963d3c720e287b13b0ccd" Nov 03 01:48:01.459581 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:01.455402 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6" Nov 03 01:48:01.635894 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:01.635856 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:48:01.641216 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:01.641175 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2b9f22c1/e2e-lifecycle-model-2b9f22c1-kserve-7dfd96649c-c89s6"] Nov 03 01:48:02.151755 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:02.151716 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:48:02.596337 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:02.596298 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" path="/var/lib/kubelet/pods/cf462ddc-c83e-4aa3-8162-9b609cef4528/volumes" Nov 03 01:48:27.878401 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:27.878366 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:48:27.878977 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:27.878616 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" containerID="cri-o://7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828" gracePeriod=30 Nov 03 01:48:27.929551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:27.929514 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:48:27.929813 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:27.929771 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" containerID="cri-o://a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b" gracePeriod=30 Nov 03 01:48:31.999254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:31.999221 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:48:31.999820 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:31.999417 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" containerID="cri-o://a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f" gracePeriod=30 Nov 03 01:48:32.007289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:32.007260 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:48:32.007541 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:32.007501 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" containerID="cri-o://75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8" gracePeriod=30 Nov 03 01:48:32.250850 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:32.250748 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:48:33.228857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.228830 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:48:33.258776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.258747 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:48:33.320409 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320378 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320435 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320458 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320475 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320506 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320528 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.320611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320604 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320747 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320775 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320757 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320867 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320905 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.320997 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321015 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321022 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321052 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321087 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321146 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321150 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321219 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321240 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321294 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\") pod \"899a899b-ccfb-4739-9f75-fe33c2954301\" (UID: \"899a899b-ccfb-4739-9f75-fe33c2954301\") " Nov 03 01:48:33.321865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321340 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.321865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321352 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321403 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\") pod \"1a064279-29a0-4c8e-9eda-8426698feba7\" (UID: \"1a064279-29a0-4c8e-9eda-8426698feba7\") " Nov 03 01:48:33.321865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321551 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.321865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.321782 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.322188 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322132 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322195 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322212 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322237 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322252 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a064279-29a0-4c8e-9eda-8426698feba7-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322266 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322281 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899a899b-ccfb-4739-9f75-fe33c2954301-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322296 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322310 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.322472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.322324 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.323822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.323788 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:33.323935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.323823 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.323935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.323890 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:33.323935 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.323888 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:33.324170 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.324151 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:33.324346 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.324330 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "kube-api-access-2mf7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:33.324414 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.324351 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv" pod "1a064279-29a0-4c8e-9eda-8426698feba7" (UID: "1a064279-29a0-4c8e-9eda-8426698feba7"). InnerVolumeSpecName "kube-api-access-cq5mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:33.324625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.324606 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token" pod "899a899b-ccfb-4739-9f75-fe33c2954301" (UID: "899a899b-ccfb-4739-9f75-fe33c2954301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:33.423484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423444 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423480 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a064279-29a0-4c8e-9eda-8426698feba7-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423494 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423507 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899a899b-ccfb-4739-9f75-fe33c2954301-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423520 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mf7f\" (UniqueName: \"kubernetes.io/projected/899a899b-ccfb-4739-9f75-fe33c2954301-kube-api-access-2mf7f\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423533 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a064279-29a0-4c8e-9eda-8426698feba7-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423544 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899a899b-ccfb-4739-9f75-fe33c2954301-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.423745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.423595 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5mv\" (UniqueName: \"kubernetes.io/projected/1a064279-29a0-4c8e-9eda-8426698feba7-kube-api-access-cq5mv\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:33.667307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.667267 4992 generic.go:356] "Generic (PLEG): container finished" podID="899a899b-ccfb-4739-9f75-fe33c2954301" containerID="a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b" exitCode=0 Nov 03 01:48:33.667497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.667314 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" Nov 03 01:48:33.667497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.667350 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" event={"ID":"899a899b-ccfb-4739-9f75-fe33c2954301","Type":"ContainerDied","Data":"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b"} Nov 03 01:48:33.667497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.667409 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r" event={"ID":"899a899b-ccfb-4739-9f75-fe33c2954301","Type":"ContainerDied","Data":"c1fc9a7a09d8f420a5fe8e611c76f5b87ea5d2d5da88f0211a4e1f3163803555"} Nov 03 01:48:33.667497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.667441 4992 scope.go:122] "RemoveContainer" containerID="a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b" Nov 03 01:48:33.669139 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.669112 4992 generic.go:356] "Generic (PLEG): container finished" podID="1a064279-29a0-4c8e-9eda-8426698feba7" containerID="7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828" exitCode=0 Nov 03 01:48:33.669247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.669150 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" Nov 03 01:48:33.669247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.669163 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" event={"ID":"1a064279-29a0-4c8e-9eda-8426698feba7","Type":"ContainerDied","Data":"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828"} Nov 03 01:48:33.669247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.669203 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7" event={"ID":"1a064279-29a0-4c8e-9eda-8426698feba7","Type":"ContainerDied","Data":"d9248dea81e0a10063028578386a4490a0610b2e18be3b0fb2400f9b3f77fe65"} Nov 03 01:48:33.679042 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.679019 4992 scope.go:122] "RemoveContainer" containerID="a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b" Nov 03 01:48:33.679366 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:33.679342 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b\": container with ID starting with a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b not found: ID does not exist" containerID="a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b" Nov 03 01:48:33.679456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.679378 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b"} err="failed to get container status \"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b\": rpc error: code = NotFound desc = could not find container \"a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b\": container with ID starting with a019702489a862bd9c819cedc43db0b9c19d5759e626844da5e59c31c980105b not found: ID does not exist" Nov 03 01:48:33.679456 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.679401 4992 scope.go:122] "RemoveContainer" containerID="7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828" Nov 03 01:48:33.690353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.690335 4992 scope.go:122] "RemoveContainer" containerID="7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828" Nov 03 01:48:33.690641 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:33.690621 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828\": container with ID starting with 7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828 not found: ID does not exist" containerID="7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828" Nov 03 01:48:33.690711 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.690651 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828"} err="failed to get container status \"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828\": rpc error: code = NotFound desc = could not find container \"7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828\": container with ID starting with 7a9f5f95a7e555e496fdb6b7366d7dfc318d0f8104a0ead06a5220e8634c0828 not found: ID does not exist" Nov 03 01:48:33.838307 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.838271 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:48:33.842572 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.842534 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-shvh7"] Nov 03 01:48:33.955227 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.955134 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:48:33.959266 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:33.959236 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-9nb7r"] Nov 03 01:48:34.562803 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.562765 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:48:34.563202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.562861 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:48:34.595781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.595745 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" path="/var/lib/kubelet/pods/1a064279-29a0-4c8e-9eda-8426698feba7/volumes" Nov 03 01:48:34.596289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.596275 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" path="/var/lib/kubelet/pods/899a899b-ccfb-4739-9f75-fe33c2954301/volumes" Nov 03 01:48:34.926647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.926610 4992 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:48:34.926817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:34.926682 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:48:37.083950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.083913 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:48:37.084397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.084161 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" containerName="maas-api" containerID="cri-o://aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d" gracePeriod=30 Nov 03 01:48:37.152661 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.152549 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.96:8443/health\": dial tcp 10.132.0.96:8443: connect: connection refused" Nov 03 01:48:37.465085 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.465051 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:48:37.492247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.486389 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:48:37.506544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.506511 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597067 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\") pod \"e8041320-c248-4ebf-84fe-d8185015d25e\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597146 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597181 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597217 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") pod \"e8041320-c248-4ebf-84fe-d8185015d25e\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597270 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597319 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597346 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597396 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597449 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597487 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597521 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597577 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\") pod \"e8041320-c248-4ebf-84fe-d8185015d25e\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597636 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597668 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\" (UniqueName: \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\") pod \"e8041320-c248-4ebf-84fe-d8185015d25e\" (UID: \"e8041320-c248-4ebf-84fe-d8185015d25e\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597734 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597764 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597808 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597876 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597922 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.597983 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\") pod \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\" (UID: \"7d7fcfd6-4c4a-425d-bbc6-940c528033ca\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.598051 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.598117 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\") pod \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\" (UID: \"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9\") " Nov 03 01:48:37.601678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.601658 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.604731 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs" pod "e8041320-c248-4ebf-84fe-d8185015d25e" (UID: "e8041320-c248-4ebf-84fe-d8185015d25e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.610899 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.611434 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.622798 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls" pod "e8041320-c248-4ebf-84fe-d8185015d25e" (UID: "e8041320-c248-4ebf-84fe-d8185015d25e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.623056 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.629005 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.647037 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:48:37.658063 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.650673 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.659817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.659249 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.659817 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.659443 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.660000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.659804 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.660000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.659841 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.660000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.659920 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.660146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.660062 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca" pod "e8041320-c248-4ebf-84fe-d8185015d25e" (UID: "e8041320-c248-4ebf-84fe-d8185015d25e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:37.660453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.660421 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:37.665484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.665227 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.665484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.665353 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:48:37.665484 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.665453 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.666417 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.666361 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4" pod "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" (UID: "b4cfa2a8-1cf1-4899-be24-29b5c9a617b9"). InnerVolumeSpecName "kube-api-access-j4mm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.666417 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.666365 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n" pod "e8041320-c248-4ebf-84fe-d8185015d25e" (UID: "e8041320-c248-4ebf-84fe-d8185015d25e"). InnerVolumeSpecName "kube-api-access-wq89n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.666745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.666723 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x" pod "7d7fcfd6-4c4a-425d-bbc6-940c528033ca" (UID: "7d7fcfd6-4c4a-425d-bbc6-940c528033ca"). InnerVolumeSpecName "kube-api-access-zcn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700401 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700433 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq89n\" (UniqueName: \"kubernetes.io/projected/e8041320-c248-4ebf-84fe-d8185015d25e-kube-api-access-wq89n\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700450 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700464 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700476 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700489 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700503 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700516 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcn4x\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-kube-api-access-zcn4x\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700530 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4mm4\" (UniqueName: \"kubernetes.io/projected/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-kube-api-access-j4mm4\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700543 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700569 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700582 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700596 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700610 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8041320-c248-4ebf-84fe-d8185015d25e-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700624 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700638 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700654 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700670 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700684 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700701 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700716 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d7fcfd6-4c4a-425d-bbc6-940c528033ca-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.700434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.700732 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8041320-c248-4ebf-84fe-d8185015d25e-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:37.703268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.703234 4992 generic.go:356] "Generic (PLEG): container finished" podID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerID="75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8" exitCode=0 Nov 03 01:48:37.703413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.703281 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" Nov 03 01:48:37.703413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.703283 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" event={"ID":"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9","Type":"ContainerDied","Data":"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8"} Nov 03 01:48:37.703413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.703335 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn" event={"ID":"b4cfa2a8-1cf1-4899-be24-29b5c9a617b9","Type":"ContainerDied","Data":"28517067756a79637fb68c3331ce853cc93b59d52c529594ab871200e120537f"} Nov 03 01:48:37.703413 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.703359 4992 scope.go:122] "RemoveContainer" containerID="75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8" Nov 03 01:48:37.705065 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.705041 4992 generic.go:356] "Generic (PLEG): container finished" podID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerID="a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f" exitCode=0 Nov 03 01:48:37.705176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.705084 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" Nov 03 01:48:37.705176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.705121 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" event={"ID":"7d7fcfd6-4c4a-425d-bbc6-940c528033ca","Type":"ContainerDied","Data":"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f"} Nov 03 01:48:37.705176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.705165 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx" event={"ID":"7d7fcfd6-4c4a-425d-bbc6-940c528033ca","Type":"ContainerDied","Data":"64cd9f0ff0d1907127818cf0a85fd0d1f8bfcd98f003f2a0fdd20b2d8eac478f"} Nov 03 01:48:37.708048 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.708024 4992 generic.go:356] "Generic (PLEG): container finished" podID="e8041320-c248-4ebf-84fe-d8185015d25e" containerID="aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d" exitCode=0 Nov 03 01:48:37.708156 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.708069 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" Nov 03 01:48:37.708156 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.708105 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" event={"ID":"e8041320-c248-4ebf-84fe-d8185015d25e","Type":"ContainerDied","Data":"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d"} Nov 03 01:48:37.708156 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.708148 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4" event={"ID":"e8041320-c248-4ebf-84fe-d8185015d25e","Type":"ContainerDied","Data":"1ca753f4c8c8ccd10a76c0c1ddce13dbe92c0a4225dff52ff31165ae841909ab"} Nov 03 01:48:37.720108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.719396 4992 scope.go:122] "RemoveContainer" containerID="75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8" Nov 03 01:48:37.720916 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:37.720886 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8\": container with ID starting with 75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8 not found: ID does not exist" containerID="75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8" Nov 03 01:48:37.721026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.720929 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8"} err="failed to get container status \"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8\": rpc error: code = NotFound desc = could not find container \"75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8\": container with ID starting with 75b9c05d093bf4dda655d0eddd75c6edd0c82b7564cd8874be72c8f676ec7ce8 not found: ID does not exist" Nov 03 01:48:37.721026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.720955 4992 scope.go:122] "RemoveContainer" containerID="a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f" Nov 03 01:48:37.732995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.732973 4992 scope.go:122] "RemoveContainer" containerID="a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f" Nov 03 01:48:37.733393 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:37.733356 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f\": container with ID starting with a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f not found: ID does not exist" containerID="a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f" Nov 03 01:48:37.733551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.733404 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f"} err="failed to get container status \"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f\": rpc error: code = NotFound desc = could not find container \"a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f\": container with ID starting with a29bf4196643618f6227781d8bb901b933fc772cf8bcb8aa4c3d952be8a9d77f not found: ID does not exist" Nov 03 01:48:37.733551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.733427 4992 scope.go:122] "RemoveContainer" containerID="aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d" Nov 03 01:48:37.749211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.749183 4992 scope.go:122] "RemoveContainer" containerID="aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d" Nov 03 01:48:37.749534 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:37.749504 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d\": container with ID starting with aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d not found: ID does not exist" containerID="aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d" Nov 03 01:48:37.749660 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:37.749543 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d"} err="failed to get container status \"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d\": rpc error: code = NotFound desc = could not find container \"aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d\": container with ID starting with aef83017eb717b648c74521f9f7d47ef90e11f23c48367c8347153d555ed994d not found: ID does not exist" Nov 03 01:48:38.020702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.020665 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:48:38.026247 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.026222 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-mtcrx"] Nov 03 01:48:38.150231 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.150094 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:48:38.153199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.152903 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-118857-694b444f78-gvdt4"] Nov 03 01:48:38.260030 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.259998 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:48:38.264317 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.264291 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-n59vn"] Nov 03 01:48:38.596266 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.596234 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" path="/var/lib/kubelet/pods/7d7fcfd6-4c4a-425d-bbc6-940c528033ca/volumes" Nov 03 01:48:38.596796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.596779 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" path="/var/lib/kubelet/pods/b4cfa2a8-1cf1-4899-be24-29b5c9a617b9/volumes" Nov 03 01:48:38.597234 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:38.597220 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" path="/var/lib/kubelet/pods/e8041320-c248-4ebf-84fe-d8185015d25e/volumes" Nov 03 01:48:59.064210 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.064174 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:48:59.064838 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.064392 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" podUID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" containerName="maas-api" containerID="cri-o://774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd" gracePeriod=30 Nov 03 01:48:59.407745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.407717 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:48:59.513471 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.513435 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\") pod \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " Nov 03 01:48:59.513665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.513492 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\") pod \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " Nov 03 01:48:59.513665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.513525 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") pod \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " Nov 03 01:48:59.513788 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.513692 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\") pod \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\" (UID: \"93e2fa35-10cf-41ce-8675-7e6b0727f5c5\") " Nov 03 01:48:59.514075 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.514045 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca" pod "93e2fa35-10cf-41ce-8675-7e6b0727f5c5" (UID: "93e2fa35-10cf-41ce-8675-7e6b0727f5c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:59.514344 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.514325 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:59.515857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.515828 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls" pod "93e2fa35-10cf-41ce-8675-7e6b0727f5c5" (UID: "93e2fa35-10cf-41ce-8675-7e6b0727f5c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:59.515950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.515874 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz" pod "93e2fa35-10cf-41ce-8675-7e6b0727f5c5" (UID: "93e2fa35-10cf-41ce-8675-7e6b0727f5c5"). InnerVolumeSpecName "kube-api-access-cfdwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:59.516087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.516062 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs" pod "93e2fa35-10cf-41ce-8675-7e6b0727f5c5" (UID: "93e2fa35-10cf-41ce-8675-7e6b0727f5c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:59.615176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.615134 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdwz\" (UniqueName: \"kubernetes.io/projected/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-kube-api-access-cfdwz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:59.615176 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.615172 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:59.615387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.615185 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e2fa35-10cf-41ce-8675-7e6b0727f5c5-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:48:59.888892 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.888844 4992 generic.go:356] "Generic (PLEG): container finished" podID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" containerID="774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd" exitCode=0 Nov 03 01:48:59.889105 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.888916 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" event={"ID":"93e2fa35-10cf-41ce-8675-7e6b0727f5c5","Type":"ContainerDied","Data":"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd"} Nov 03 01:48:59.889105 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.888963 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" Nov 03 01:48:59.889105 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.888987 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr" event={"ID":"93e2fa35-10cf-41ce-8675-7e6b0727f5c5","Type":"ContainerDied","Data":"b29dc739a1355501ca1d706c2521d0857006d30dfc582b8536f333469af21e94"} Nov 03 01:48:59.889105 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.889013 4992 scope.go:122] "RemoveContainer" containerID="774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd" Nov 03 01:48:59.900551 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.900525 4992 scope.go:122] "RemoveContainer" containerID="774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd" Nov 03 01:48:59.900870 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:48:59.900849 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd\": container with ID starting with 774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd not found: ID does not exist" containerID="774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd" Nov 03 01:48:59.900932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:48:59.900879 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd"} err="failed to get container status \"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd\": rpc error: code = NotFound desc = could not find container \"774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd\": container with ID starting with 774d946267aa7ca3e21957434f0c89d5e2e4047ac125cc2e09b5a0db2b0c27cd not found: ID does not exist" Nov 03 01:49:00.063099 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:00.063048 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:49:00.113890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:00.113848 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2b9f22c1-575597b4f9-ts7jr"] Nov 03 01:49:00.595701 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:00.595666 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" path="/var/lib/kubelet/pods/93e2fa35-10cf-41ce-8675-7e6b0727f5c5/volumes" Nov 03 01:49:02.153128 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:49:02.153088 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:49:09.516386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.516349 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:49:09.516830 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.516593 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" containerID="cri-o://851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab" gracePeriod=30 Nov 03 01:49:09.529188 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.529158 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:49:09.529381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.529346 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" containerID="cri-o://b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02" gracePeriod=30 Nov 03 01:49:09.545825 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.545794 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:49:09.546050 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:09.546013 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" containerID="cri-o://f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816" gracePeriod=30 Nov 03 01:49:10.843514 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.843472 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:49:10.844265 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844242 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844268 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844289 4992 cpu_manager.go:414] "containerMap: removing container" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844296 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844330 4992 cpu_manager.go:414] "containerMap: removing container" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844338 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844346 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" containerName="maas-api" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844354 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" containerName="maas-api" Nov 03 01:49:10.844372 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844371 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844379 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844388 4992 cpu_manager.go:414] "containerMap: removing container" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844396 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844413 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844421 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844441 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844449 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844460 4992 cpu_manager.go:414] "containerMap: removing container" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerName="main" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844468 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerName="main" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844486 4992 cpu_manager.go:414] "containerMap: removing container" podUID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" containerName="maas-api" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844494 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" containerName="maas-api" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844660 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="899a899b-ccfb-4739-9f75-fe33c2954301" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844676 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8041320-c248-4ebf-84fe-d8185015d25e" containerName="maas-api" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844690 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cfa2a8-1cf1-4899-be24-29b5c9a617b9" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844699 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7fcfd6-4c4a-425d-bbc6-940c528033ca" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844709 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844719 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844728 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b336bf7a-8c4a-45e9-b7a4-93505278f1a4" containerName="istio-proxy" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844737 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf462ddc-c83e-4aa3-8162-9b609cef4528" containerName="main" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844748 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e2fa35-10cf-41ce-8675-7e6b0727f5c5" containerName="maas-api" Nov 03 01:49:10.844812 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.844762 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a064279-29a0-4c8e-9eda-8426698feba7" containerName="istio-proxy" Nov 03 01:49:10.848640 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.848619 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.919154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.919118 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:49:10.966656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.966621 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e0c1fb-openshift-default-dockercfg-sp9nh\"" Nov 03 01:49:10.975987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.975949 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976011 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzpl\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976108 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976155 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976192 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976279 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976314 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976378 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:10.976579 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:10.976517 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.080488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.080443 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.080714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.080528 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.080714 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.080636 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzpl\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.081281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.081255 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082143 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082117 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082186 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082222 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082362 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082286 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082362 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082335 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.082470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.082412 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.083161 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.083132 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.083400 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.083377 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.084336 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.084311 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.084603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.084581 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.104845 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.099308 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.114377 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.110654 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.163608 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.163550 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.163775 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.163721 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzpl\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\") pod \"e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.463159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.463123 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:11.895200 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:49:11.895168 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299bbb0f_8ef7_4707_8cc8_be151d796a9b.slice/crio-8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49 WatchSource:0}: Error finding container 8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49: Status 404 returned error can't find the container with id 8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49 Nov 03 01:49:11.898088 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.898062 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:49:11.898226 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.898193 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:11.898294 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.898262 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:11.898351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:11.898306 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:12.035531 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:12.035483 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerStarted","Data":"f1536425119e9972e312c2f0629cb0660893187adbbc3875f8d0a61aecc61876"} Nov 03 01:49:12.035734 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:12.035661 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerStarted","Data":"8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49"} Nov 03 01:49:12.162202 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:12.162091 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podStartSLOduration=2.162074851 podStartE2EDuration="2.162074851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:12.159911705 +0000 UTC m=+1887.738734291" watchObservedRunningTime="2026-11-03 01:49:12.162074851 +0000 UTC m=+1887.740897438" Nov 03 01:49:12.464011 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:12.463973 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:12.469797 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:12.469767 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:13.043196 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:13.043162 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:13.044069 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:13.044051 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:49:14.841200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.841167 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:49:14.843702 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843673 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.843844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843767 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.843844 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843812 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.843958 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843861 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.843958 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843920 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.843958 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843949 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.844117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843981 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.844117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.843994 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.844117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844068 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.844279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844113 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\") pod \"055c1fe3-3f0d-46d0-b238-7f007eed1591\" (UID: \"055c1fe3-3f0d-46d0-b238-7f007eed1591\") " Nov 03 01:49:14.844279 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844206 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.844739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844417 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.844739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844619 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.844739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844642 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.844739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.844657 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.845231 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.845152 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.846041 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.846014 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.848329 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.848305 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.848608 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.848570 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.849620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.849595 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "kube-api-access-qqspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.849976 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.849951 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy" pod "055c1fe3-3f0d-46d0-b238-7f007eed1591" (UID: "055c1fe3-3f0d-46d0-b238-7f007eed1591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.863447 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.863420 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:49:14.874799 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.874764 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:49:14.945132 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945094 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945165 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945190 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945314 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945375 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.945438 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945413 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945467 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.945621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945512 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.945621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945550 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945621 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945588 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945623 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945693 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945753 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945764 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945791 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.945829 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945800 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945839 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945962 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\") pod \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\" (UID: \"1c8ff482-4a35-4e2d-b6a2-6005c07d128c\") " Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945999 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946051 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946078 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\") pod \"420e913d-1162-402f-9234-90c59e4f87cb\" (UID: \"420e913d-1162-402f-9234-90c59e4f87cb\") " Nov 03 01:49:14.946166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945992 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946287 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946311 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946448 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946357 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:14.946636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.945936 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946837 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946814 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.946946 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.946886 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.947796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947764 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947802 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqspp\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-kube-api-access-qqspp\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947820 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947835 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947848 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947860 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947871 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947888 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.947905 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947903 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947919 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947935 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947948 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055c1fe3-3f0d-46d0-b238-7f007eed1591-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947963 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420e913d-1162-402f-9234-90c59e4f87cb-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947979 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055c1fe3-3f0d-46d0-b238-7f007eed1591-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.947989 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.948365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.948275 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "kube-api-access-qj8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.948726 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.948574 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "kube-api-access-hmkbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.948782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.948765 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:14.948884 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.948864 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.949209 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.949189 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo" pod "1c8ff482-4a35-4e2d-b6a2-6005c07d128c" (UID: "1c8ff482-4a35-4e2d-b6a2-6005c07d128c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:49:14.949387 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.949358 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:49:14.949678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:14.949654 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token" pod "420e913d-1162-402f-9234-90c59e4f87cb" (UID: "420e913d-1162-402f-9234-90c59e4f87cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049800 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8xn\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-kube-api-access-qj8xn\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049836 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049852 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420e913d-1162-402f-9234-90c59e4f87cb-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049865 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049881 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.049890 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049894 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.050908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049903 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420e913d-1162-402f-9234-90c59e4f87cb-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.050908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049913 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420e913d-1162-402f-9234-90c59e4f87cb-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.050908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049923 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.050908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.049936 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmkbg\" (UniqueName: \"kubernetes.io/projected/1c8ff482-4a35-4e2d-b6a2-6005c07d128c-kube-api-access-hmkbg\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:49:15.057510 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.057478 4992 generic.go:356] "Generic (PLEG): container finished" podID="420e913d-1162-402f-9234-90c59e4f87cb" containerID="851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab" exitCode=0 Nov 03 01:49:15.057677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.057539 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" Nov 03 01:49:15.057677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.057576 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" event={"ID":"420e913d-1162-402f-9234-90c59e4f87cb","Type":"ContainerDied","Data":"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab"} Nov 03 01:49:15.057677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.057638 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds" event={"ID":"420e913d-1162-402f-9234-90c59e4f87cb","Type":"ContainerDied","Data":"8bcc4d8c9e58a48c7e23d2ffd6fd815a75b37c4bf6d5b4e32c773026edbcf2ca"} Nov 03 01:49:15.057677 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.057660 4992 scope.go:122] "RemoveContainer" containerID="851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab" Nov 03 01:49:15.059177 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.059146 4992 generic.go:356] "Generic (PLEG): container finished" podID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerID="b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02" exitCode=0 Nov 03 01:49:15.059293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.059194 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" Nov 03 01:49:15.059293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.059229 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" event={"ID":"055c1fe3-3f0d-46d0-b238-7f007eed1591","Type":"ContainerDied","Data":"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02"} Nov 03 01:49:15.059293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.059284 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl" event={"ID":"055c1fe3-3f0d-46d0-b238-7f007eed1591","Type":"ContainerDied","Data":"ac43a6cb63d47bb5729347de3ea06928d1ee6ed4025694950b07786d485d77a1"} Nov 03 01:49:15.060965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.060947 4992 generic.go:356] "Generic (PLEG): container finished" podID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerID="f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816" exitCode=0 Nov 03 01:49:15.061074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.061032 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" Nov 03 01:49:15.061074 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.061066 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" event={"ID":"1c8ff482-4a35-4e2d-b6a2-6005c07d128c","Type":"ContainerDied","Data":"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816"} Nov 03 01:49:15.061192 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.061102 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj" event={"ID":"1c8ff482-4a35-4e2d-b6a2-6005c07d128c","Type":"ContainerDied","Data":"47c8ad2d5f38e3f94ba495e4cc2a5f2d2169ba91c59f7fd02666a06e5ef31619"} Nov 03 01:49:15.071078 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.071055 4992 scope.go:122] "RemoveContainer" containerID="851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab" Nov 03 01:49:15.071356 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:49:15.071334 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab\": container with ID starting with 851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab not found: ID does not exist" containerID="851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab" Nov 03 01:49:15.071440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.071367 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab"} err="failed to get container status \"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab\": rpc error: code = NotFound desc = could not find container \"851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab\": container with ID starting with 851b7b6e81daf6c14507869408d7bdf5254c48a8b28aace87ea676fae3f143ab not found: ID does not exist" Nov 03 01:49:15.071440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.071391 4992 scope.go:122] "RemoveContainer" containerID="b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02" Nov 03 01:49:15.083865 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.083843 4992 scope.go:122] "RemoveContainer" containerID="b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02" Nov 03 01:49:15.084186 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:49:15.084164 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02\": container with ID starting with b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02 not found: ID does not exist" containerID="b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02" Nov 03 01:49:15.084229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.084195 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02"} err="failed to get container status \"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02\": rpc error: code = NotFound desc = could not find container \"b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02\": container with ID starting with b11e3f07983ac4e3578d39ae66e3d012925d46df7f72888863a7fe99f3b2ea02 not found: ID does not exist" Nov 03 01:49:15.084229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.084213 4992 scope.go:122] "RemoveContainer" containerID="f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816" Nov 03 01:49:15.093223 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.093201 4992 scope.go:122] "RemoveContainer" containerID="f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816" Nov 03 01:49:15.093530 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:49:15.093505 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816\": container with ID starting with f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816 not found: ID does not exist" containerID="f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816" Nov 03 01:49:15.093630 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.093543 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816"} err="failed to get container status \"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816\": rpc error: code = NotFound desc = could not find container \"f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816\": container with ID starting with f01cce6110b0fd0165396fcf4cba9a734057931a85021793ee27b71b59166816 not found: ID does not exist" Nov 03 01:49:15.384136 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.384046 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:49:15.387744 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.387713 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-27vhj"] Nov 03 01:49:15.502220 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.502185 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:49:15.507572 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.507527 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-rk6ds"] Nov 03 01:49:15.622577 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.622526 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:49:15.626261 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:15.626232 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-nb6cl"] Nov 03 01:49:16.595423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:16.595379 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" path="/var/lib/kubelet/pods/055c1fe3-3f0d-46d0-b238-7f007eed1591/volumes" Nov 03 01:49:16.596096 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:16.596074 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" path="/var/lib/kubelet/pods/1c8ff482-4a35-4e2d-b6a2-6005c07d128c/volumes" Nov 03 01:49:16.596598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:16.596574 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420e913d-1162-402f-9234-90c59e4f87cb" path="/var/lib/kubelet/pods/420e913d-1162-402f-9234-90c59e4f87cb/volumes" Nov 03 01:49:32.150913 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:49:32.150869 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:49:41.109415 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.109372 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 01:49:41.110200 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110175 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:41.110249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110205 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:41.110249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110233 4992 cpu_manager.go:414] "containerMap: removing container" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" Nov 03 01:49:41.110249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110241 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" Nov 03 01:49:41.110341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110262 4992 cpu_manager.go:414] "containerMap: removing container" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" Nov 03 01:49:41.110341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110270 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" Nov 03 01:49:41.110341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110317 4992 cpu_manager.go:414] "containerMap: removing container" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" Nov 03 01:49:41.110341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110324 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" Nov 03 01:49:41.110476 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110464 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="055c1fe3-3f0d-46d0-b238-7f007eed1591" containerName="istio-proxy" Nov 03 01:49:41.110511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110481 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8ff482-4a35-4e2d-b6a2-6005c07d128c" containerName="istio-proxy" Nov 03 01:49:41.110511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110498 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b442232f-a4b4-459a-b851-63203d3d9bce" containerName="istio-proxy" Nov 03 01:49:41.110511 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.110508 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="420e913d-1162-402f-9234-90c59e4f87cb" containerName="istio-proxy" Nov 03 01:49:41.114353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.114334 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.228690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.228656 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b177c9-openshift-default-dockercfg-frfs4\"" Nov 03 01:49:41.258542 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.240242 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.307864 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.307931 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.307973 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308006 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnpw2\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308042 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308073 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308155 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308233 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.308594 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.308334 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.409647 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.409534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.409847 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.409685 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410250 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410220 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410362 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410314 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410419 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410369 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410475 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410413 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnpw2\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410475 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410462 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.410589 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410505 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.411750 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.411109 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.412023 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.411993 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.412740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.412523 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.412839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.412765 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.412839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.412806 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.412956 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.410585 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.417268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.415292 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.444382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.444348 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.474390 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.474354 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.474595 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.474478 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnpw2\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\") pod \"e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:41.728364 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:41.728324 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:42.156710 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:49:42.156678 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22fd95a_ae21_4278_bc3d_3e9e5afb1298.slice/crio-6dc01b296c8aa85ab8f0a9fc0c332fcbe5443d0dfdcefd81f9276b0cd509fda5 WatchSource:0}: Error finding container 6dc01b296c8aa85ab8f0a9fc0c332fcbe5443d0dfdcefd81f9276b0cd509fda5: Status 404 returned error can't find the container with id 6dc01b296c8aa85ab8f0a9fc0c332fcbe5443d0dfdcefd81f9276b0cd509fda5 Nov 03 01:49:42.158986 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.158963 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 01:49:42.159955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.159916 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:42.160081 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.160000 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:42.160081 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.160044 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:49:42.283619 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.283068 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerStarted","Data":"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10"} Nov 03 01:49:42.283619 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.283133 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerStarted","Data":"6dc01b296c8aa85ab8f0a9fc0c332fcbe5443d0dfdcefd81f9276b0cd509fda5"} Nov 03 01:49:42.408709 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.408616 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podStartSLOduration=1.408599282 podStartE2EDuration="1.408599282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:42.406708631 +0000 UTC m=+1917.985531218" watchObservedRunningTime="2026-11-03 01:49:42.408599282 +0000 UTC m=+1917.987421866" Nov 03 01:49:42.728618 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:42.728581 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:43.734292 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:43.734260 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:44.295903 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:44.295871 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:49:44.296897 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:49:44.296878 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:50:02.150401 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:02.150317 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:50:04.970939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:04.970906 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:50:04.976393 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:04.976375 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.029995 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.029959 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:50:05.078614 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.078550 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e0c1fb-serving-cert\"" Nov 03 01:50:05.127412 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.127378 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.127611 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.127432 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.127690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.127636 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.127690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.127676 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.228685 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.228588 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.228685 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.228677 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.228926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.228698 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.228926 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.228833 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.230090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.230062 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.231228 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.231199 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.231325 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.231292 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.293327 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.293286 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\") pod \"maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.589775 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.589684 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:05.964024 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:50:05.963990 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf47964_5aeb_4fa2_a73f_1a353b3885a2.slice/crio-c44f0dd1c83edab0c9e0dfa5d5bd70f27ee6e79fe529d7cdea6a1752b03cc388 WatchSource:0}: Error finding container c44f0dd1c83edab0c9e0dfa5d5bd70f27ee6e79fe529d7cdea6a1752b03cc388: Status 404 returned error can't find the container with id c44f0dd1c83edab0c9e0dfa5d5bd70f27ee6e79fe529d7cdea6a1752b03cc388 Nov 03 01:50:05.966778 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:05.966748 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:50:06.438201 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:06.438162 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" event={"ID":"4bf47964-5aeb-4fa2-a73f-1a353b3885a2","Type":"ContainerStarted","Data":"c44f0dd1c83edab0c9e0dfa5d5bd70f27ee6e79fe529d7cdea6a1752b03cc388"} Nov 03 01:50:07.454794 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:07.454758 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" event={"ID":"4bf47964-5aeb-4fa2-a73f-1a353b3885a2","Type":"ContainerStarted","Data":"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7"} Nov 03 01:50:07.455192 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:07.454858 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:07.577377 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:07.577319 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" podStartSLOduration=2.728247143 podStartE2EDuration="3.577306767s" totalImagesPullingTime="849.059624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:07.575232707 +0000 UTC m=+1943.154055293" watchObservedRunningTime="2026-11-03 01:50:07.577306767 +0000 UTC m=+1943.156129389" Nov 03 01:50:13.467743 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:13.467709 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:50:19.151628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:19.151594 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:50:19.152121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:19.151878 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" containerID="cri-o://a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235" gracePeriod=30 Nov 03 01:50:21.372638 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.372588 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:50:21.378546 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.378520 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.437617 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.437584 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-71746a-openshift-default-dockercfg-vnj9h\"" Nov 03 01:50:21.444462 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.444422 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.444783 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.444759 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.444948 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.444923 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.445090 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.445074 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.445237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.445221 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.445375 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.445360 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.445616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.445597 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.445954 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.445934 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.446092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.446074 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhw88\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.506472 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.504552 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:50:21.573880 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.550241 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.574284 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.558847 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.574422 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.574253 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575351 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575318 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575481 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575393 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhw88\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575542 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575508 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575618 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575597 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575679 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575661 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575704 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575789 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575761 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.575841 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.575806 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.576309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.576282 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.576966 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.576939 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.577539 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.577509 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.599603 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.587348 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.632623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.632467 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.641969 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.641933 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.642151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.642055 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhw88\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\") pod \"e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:21.692952 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:21.692915 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:22.137281 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:50:22.137243 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1bcd02_c463_48b1_ba8f_6c47464a0feb.slice/crio-ff842564f8870a7a92ddb1c6dc842a1f1c20415c62e96638df3b8fe386ab559c WatchSource:0}: Error finding container ff842564f8870a7a92ddb1c6dc842a1f1c20415c62e96638df3b8fe386ab559c: Status 404 returned error can't find the container with id ff842564f8870a7a92ddb1c6dc842a1f1c20415c62e96638df3b8fe386ab559c Nov 03 01:50:22.139821 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.139795 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:50:22.140154 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.140116 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:22.140222 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.140204 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:22.140269 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.140254 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:22.612783 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.612749 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerStarted","Data":"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735"} Nov 03 01:50:22.613187 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.612802 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerStarted","Data":"ff842564f8870a7a92ddb1c6dc842a1f1c20415c62e96638df3b8fe386ab559c"} Nov 03 01:50:22.693308 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.693268 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:22.739243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:22.739185 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podStartSLOduration=1.739170297 podStartE2EDuration="1.739170297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:22.736365401 +0000 UTC m=+1958.315187988" watchObservedRunningTime="2026-11-03 01:50:22.739170297 +0000 UTC m=+1958.317992886" Nov 03 01:50:23.699148 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:23.699120 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:24.009249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.009174 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:50:24.009249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.009227 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:50:24.425631 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.425606 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:50:24.516970 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.516934 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.516985 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517019 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517061 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517164 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517137 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517177 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517208 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517239 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517261 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:24.517381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517304 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\") pod \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\" (UID: \"dfd55298-7ed7-49ff-ad1a-38c66cae9992\") " Nov 03 01:50:24.517687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517447 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:24.517687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517467 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:24.517687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517525 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:24.517809 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.517719 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:24.518043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.518019 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.518133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.518048 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.518133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.518068 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.518133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.518082 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.518133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.518096 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.519452 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.519428 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:50:24.519544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.519430 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:24.519544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.519468 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:50:24.519656 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.519638 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl" pod "dfd55298-7ed7-49ff-ad1a-38c66cae9992" (UID: "dfd55298-7ed7-49ff-ad1a-38c66cae9992"). InnerVolumeSpecName "kube-api-access-6gdvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:24.619682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.619651 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.619682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.619678 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.619682 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.619688 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/dfd55298-7ed7-49ff-ad1a-38c66cae9992-kube-api-access-6gdvl\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.619964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.619697 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd55298-7ed7-49ff-ad1a-38c66cae9992-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:50:24.625806 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.625775 4992 generic.go:356] "Generic (PLEG): container finished" podID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerID="a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235" exitCode=0 Nov 03 01:50:24.625947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.625826 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" Nov 03 01:50:24.625947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.625860 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" event={"ID":"dfd55298-7ed7-49ff-ad1a-38c66cae9992","Type":"ContainerDied","Data":"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235"} Nov 03 01:50:24.625947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.625913 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5" event={"ID":"dfd55298-7ed7-49ff-ad1a-38c66cae9992","Type":"ContainerDied","Data":"d44b36956d1e1b9bdd2d271b7d7adbf5c937a18a72c1a867e85659c18e02c474"} Nov 03 01:50:24.625947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.625937 4992 scope.go:122] "RemoveContainer" containerID="a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235" Nov 03 01:50:24.626249 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.626213 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:24.636205 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.636181 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:50:24.645497 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.645478 4992 scope.go:122] "RemoveContainer" containerID="a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235" Nov 03 01:50:24.645841 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:24.645817 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235\": container with ID starting with a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235 not found: ID does not exist" containerID="a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235" Nov 03 01:50:24.645910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.645850 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235"} err="failed to get container status \"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235\": rpc error: code = NotFound desc = could not find container \"a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235\": container with ID starting with a4f97a3d6ba0fb9e9cbd0b01eef357f6ee3ad67d653c094e6e8ec5dc59b19235 not found: ID does not exist" Nov 03 01:50:24.812293 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.811597 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:50:24.815858 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:24.815824 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-xt6t5"] Nov 03 01:50:26.228204 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.228166 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:50:26.228764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.228748 4992 cpu_manager.go:414] "containerMap: removing container" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" Nov 03 01:50:26.228811 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.228765 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" Nov 03 01:50:26.228897 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.228887 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" containerName="istio-proxy" Nov 03 01:50:26.233834 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.233815 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.290552 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.290517 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:50:26.338684 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.338646 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-86e679-serving-cert\"" Nov 03 01:50:26.341404 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.341367 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.341588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.341495 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.341653 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.341584 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j4k\" (UniqueName: \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.341746 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.341717 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.443598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.443534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.443787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.443627 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4k\" (UniqueName: \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.443787 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.443740 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.443911 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.443846 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.443947 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:26.443908 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-serving-cert: secret "maas-api-e2e-ipp-b-86e679-serving-cert" not found Nov 03 01:50:26.443999 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:26.443982 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls podName:ca74b3a0-252b-44ce-8833-98d4519fcbc9 nodeName:}" failed. No retries permitted until 2026-11-03 01:50:26.943962187 +0000 UTC m=+1962.522784772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls") pod "maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" (UID: "ca74b3a0-252b-44ce-8833-98d4519fcbc9") : secret "maas-api-e2e-ipp-b-86e679-serving-cert" not found Nov 03 01:50:26.444470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.444441 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.446109 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.446087 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.520276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.520187 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j4k\" (UniqueName: \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.601385 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.601337 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd55298-7ed7-49ff-ad1a-38c66cae9992" path="/var/lib/kubelet/pods/dfd55298-7ed7-49ff-ad1a-38c66cae9992/volumes" Nov 03 01:50:26.953277 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.953242 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:26.956027 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:26.955991 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:27.147548 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:27.147516 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:27.557339 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:50:27.557303 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca74b3a0_252b_44ce_8833_98d4519fcbc9.slice/crio-a3ea0f9ee86839fb578f21701f58473295992e0769e5b1e039cc931829d9ab84 WatchSource:0}: Error finding container a3ea0f9ee86839fb578f21701f58473295992e0769e5b1e039cc931829d9ab84: Status 404 returned error can't find the container with id a3ea0f9ee86839fb578f21701f58473295992e0769e5b1e039cc931829d9ab84 Nov 03 01:50:27.560485 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:27.560459 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:50:27.648470 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:27.648435 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" event={"ID":"ca74b3a0-252b-44ce-8833-98d4519fcbc9","Type":"ContainerStarted","Data":"a3ea0f9ee86839fb578f21701f58473295992e0769e5b1e039cc931829d9ab84"} Nov 03 01:50:29.743259 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:29.743216 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" event={"ID":"ca74b3a0-252b-44ce-8833-98d4519fcbc9","Type":"ContainerStarted","Data":"cba122d54a1557d6bbbf4fb3f375186a518db374eab5fb4e769608e1c9a8cca5"} Nov 03 01:50:29.743752 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:29.743727 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:29.870268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:29.870199 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" podStartSLOduration=2.881940548 podStartE2EDuration="3.870177887s" totalImagesPullingTime="988.237339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:29.865821176 +0000 UTC m=+1965.444643766" watchObservedRunningTime="2026-11-03 01:50:29.870177887 +0000 UTC m=+1965.449000474" Nov 03 01:50:32.152065 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:32.152021 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:50:34.777869 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.777828 4992 generic.go:356] "Generic (PLEG): container finished" podID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerID="f1536425119e9972e312c2f0629cb0660893187adbbc3875f8d0a61aecc61876" exitCode=0 Nov 03 01:50:34.778300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.777892 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerDied","Data":"f1536425119e9972e312c2f0629cb0660893187adbbc3875f8d0a61aecc61876"} Nov 03 01:50:34.778357 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.778342 4992 scope.go:122] "RemoveContainer" containerID="f1536425119e9972e312c2f0629cb0660893187adbbc3875f8d0a61aecc61876" Nov 03 01:50:34.779921 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.779892 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:34.779988 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.779961 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:34.780031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:34.779993 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:35.757093 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:35.757064 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:50:35.793632 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:35.793598 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerStarted","Data":"2306cd1b57099d0c2d263befb4e204a9f2dbec6109bf88eb229a9ed2a1cca29e"} Nov 03 01:50:36.483582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:36.471016 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:50:36.483582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:36.471170 4992 patch_prober.go:29] interesting pod/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 01:50:36.483582 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:36.471236 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 01:50:37.484012 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:37.483954 4992 patch_prober.go:29] interesting pod/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 01:50:37.486703 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:37.486659 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 01:50:38.474030 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:38.473989 4992 patch_prober.go:29] interesting pod/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 01:50:38.474325 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:38.474289 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 01:50:39.486782 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:39.486712 4992 patch_prober.go:29] interesting pod/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 01:50:39.487199 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:39.486810 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 01:50:40.483585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:40.476994 4992 patch_prober.go:29] interesting pod/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 01:50:40.483585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:40.477060 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 01:50:41.467672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:41.467643 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:50:41.468087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:41.468067 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:50:41.468861 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:41.468839 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:50:46.082943 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.082899 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:50:46.088938 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.088911 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.182751 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.182703 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.182945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.182777 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.182945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.182885 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.182945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.182935 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.183107 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.183077 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrxz\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.183165 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.183115 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.183165 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.183138 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.183268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.183167 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.183268 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.183263 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.195345 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.195319 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b37ba5-openshift-default-dockercfg-djmkx\"" Nov 03 01:50:46.203115 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.203082 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:50:46.284359 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.284314 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 01:50:46.284822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.284794 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.284945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.284844 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.284945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.284906 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285066 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.284974 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285066 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285025 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285068 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285168 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285110 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285280 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285221 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrxz\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285280 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285259 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285386 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285274 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285695 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.285937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285913 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.286014 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.285947 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.286526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.286505 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.288650 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.288624 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.288857 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.288836 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.290166 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.290144 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.348983 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.348420 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b177c9-serving-cert\"" Nov 03 01:50:46.373288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.363327 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrxz\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.373288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.370337 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 01:50:46.373288 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.373207 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\") pod \"e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.389211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.388772 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.389211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.388837 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.389211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.388947 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.389211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.389029 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6vb\" (UniqueName: \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.405381 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.405345 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:46.490802 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.490729 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.490916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.490864 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6vb\" (UniqueName: \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.491052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.491029 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.491130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.491079 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.492493 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:46.491838 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-serving-cert: secret "maas-api-e2e-ipp-a-b177c9-serving-cert" not found Nov 03 01:50:46.492493 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:50:46.491894 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls podName:0ef0bc65-ba8a-4e7f-878f-530c05131b6d nodeName:}" failed. No retries permitted until 2026-11-03 01:50:46.991879823 +0000 UTC m=+1982.570702402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls") pod "maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" (UID: "0ef0bc65-ba8a-4e7f-878f-530c05131b6d") : secret "maas-api-e2e-ipp-a-b177c9-serving-cert" not found Nov 03 01:50:46.495219 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.494893 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.495219 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.495142 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.556352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.556313 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6vb\" (UniqueName: \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:46.880766 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.876583 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:50:46.885568 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.885500 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:46.885699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.885619 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:46.885699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.885667 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:50:46.915792 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:46.915758 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerStarted","Data":"aa3588ae05720b7824159740adafad81b914e408a517a3a6807a7c3234838c93"} Nov 03 01:50:47.009457 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.009423 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:47.012043 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.012018 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:47.279976 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.279940 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:47.673884 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:50:47.673851 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef0bc65_ba8a_4e7f_878f_530c05131b6d.slice/crio-90a7a982772fc16a46dd9ce9c6bc02c879ae3cd70812c9697c5e177d655c81ad WatchSource:0}: Error finding container 90a7a982772fc16a46dd9ce9c6bc02c879ae3cd70812c9697c5e177d655c81ad: Status 404 returned error can't find the container with id 90a7a982772fc16a46dd9ce9c6bc02c879ae3cd70812c9697c5e177d655c81ad Nov 03 01:50:47.676960 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.676937 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 01:50:47.923000 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.922909 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" event={"ID":"0ef0bc65-ba8a-4e7f-878f-530c05131b6d","Type":"ContainerStarted","Data":"90a7a982772fc16a46dd9ce9c6bc02c879ae3cd70812c9697c5e177d655c81ad"} Nov 03 01:50:47.924612 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:47.924588 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerStarted","Data":"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9"} Nov 03 01:50:48.050433 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.050373 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podStartSLOduration=2.050355597 podStartE2EDuration="2.050355597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:48.048849976 +0000 UTC m=+1983.627672577" watchObservedRunningTime="2026-11-03 01:50:48.050355597 +0000 UTC m=+1983.629178188" Nov 03 01:50:48.406421 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.406327 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:48.412616 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.412550 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:48.932663 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.932621 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" event={"ID":"0ef0bc65-ba8a-4e7f-878f-530c05131b6d","Type":"ContainerStarted","Data":"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c"} Nov 03 01:50:48.932839 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.932692 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:50:48.932893 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.932866 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:48.933777 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:48.933758 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:50:49.055604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:49.055489 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" podStartSLOduration=2.094912798 podStartE2EDuration="3.055476402s" totalImagesPullingTime="960.563604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:49.053364423 +0000 UTC m=+1984.632187009" watchObservedRunningTime="2026-11-03 01:50:49.055476402 +0000 UTC m=+1984.634298989" Nov 03 01:50:54.965585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:50:54.958442 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 01:51:02.151713 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:51:02.151667 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:51:15.338724 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:15.338684 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:51:15.339162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:15.338752 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:51:17.149999 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.149961 4992 generic.go:356] "Generic (PLEG): container finished" podID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerID="e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10" exitCode=0 Nov 03 01:51:17.150440 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.150000 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerDied","Data":"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10"} Nov 03 01:51:17.150503 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.150487 4992 scope.go:122] "RemoveContainer" containerID="e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10" Nov 03 01:51:17.152208 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.152172 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:17.152300 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.152259 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:17.152340 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:17.152298 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:18.159001 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:18.158965 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerStarted","Data":"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192"} Nov 03 01:51:18.729553 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:18.729507 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:51:18.729824 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:18.729793 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:18.729964 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:18.729850 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:19.729522 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:19.729483 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:19.729955 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:19.729546 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:20.729454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:20.729408 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:20.729664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:20.729472 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:21.729254 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:21.729213 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:21.729449 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:21.729271 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:22.728897 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:22.728850 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:22.729398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:22.728922 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:23.728754 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:23.728716 4992 patch_prober.go:29] interesting pod/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 01:51:23.728944 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:23.728790 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 01:51:24.733736 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:24.733648 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:51:24.734136 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:24.734114 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:51:24.734736 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:24.734719 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 01:51:32.152451 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:51:32.152411 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:51:53.486588 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.486540 4992 generic.go:356] "Generic (PLEG): container finished" podID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerID="766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735" exitCode=0 Nov 03 01:51:53.487036 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.486613 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerDied","Data":"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735"} Nov 03 01:51:53.487097 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.487075 4992 scope.go:122] "RemoveContainer" containerID="766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735" Nov 03 01:51:53.488662 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.488634 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:53.488721 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.488706 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:53.488757 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:53.488738 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:51:54.494684 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:54.494644 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerStarted","Data":"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505"} Nov 03 01:51:54.626947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:54.626912 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:51:54.694028 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:54.693983 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:51:54.695141 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:54.695110 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:51:54.695276 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:54.695162 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 01:51:55.693495 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:55.693456 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:51:55.693977 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:55.693522 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 01:51:56.694759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:56.694717 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:51:56.695278 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:56.694819 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 01:51:57.693690 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:57.693655 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:51:57.693917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:57.693729 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 01:51:58.693729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:58.693684 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 01:51:58.694147 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:58.693773 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 01:51:59.747583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:59.747531 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:51:59.749242 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:51:59.749220 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:52:02.151017 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:52:02.150984 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:52:32.152178 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:52:32.152137 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:52:37.776764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.776670 4992 generic.go:356] "Generic (PLEG): container finished" podID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerID="e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9" exitCode=0 Nov 03 01:52:37.776764 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.776748 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerDied","Data":"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9"} Nov 03 01:52:37.777257 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.777241 4992 scope.go:122] "RemoveContainer" containerID="e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9" Nov 03 01:52:37.778848 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.778816 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:52:37.778945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.778886 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:52:37.778945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:37.778922 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:52:38.787086 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:38.787050 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerStarted","Data":"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0"} Nov 03 01:52:39.405815 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:39.405771 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:52:39.406033 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:39.406004 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:39.406104 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:39.406082 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:40.406155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:40.406115 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:40.406628 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:40.406196 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:41.406424 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:41.406385 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:41.406881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:41.406446 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:42.405870 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:42.405834 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:42.406031 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:42.405887 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:43.406370 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:43.406323 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:43.406917 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:43.406395 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:44.406026 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:44.405988 4992 patch_prober.go:29] interesting pod/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 01:52:44.406212 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:44.406046 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 01:52:44.839881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:44.839854 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:52:44.844974 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:44.844947 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:52:45.410687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:45.410655 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:52:45.410953 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:45.410933 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:52:45.411640 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:52:45.411625 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:53:02.151933 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:53:02.151844 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:53:32.151920 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:53:32.151874 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:54:00.333507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:00.333458 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:54:00.334117 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:00.333822 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" containerID="cri-o://ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be" gracePeriod=30 Nov 03 01:54:02.115157 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:02.115117 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:54:02.115577 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:02.115189 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:54:02.151760 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:54:02.151723 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:54:05.615887 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.615853 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:54:05.636770 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.636732 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.636987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.636811 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.636987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.636846 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.636987 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.636929 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.637172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.637053 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.637172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.637130 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.639474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.637205 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.639474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.637274 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.639474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.637304 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\") pod \"e664a08e-94c9-442b-b835-81ee0ea0908e\" (UID: \"e664a08e-94c9-442b-b835-81ee0ea0908e\") " Nov 03 01:54:05.639474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.638547 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:05.639474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.639179 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:05.640605 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.640399 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:05.641172 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.641148 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:05.641402 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.641347 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:05.700620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.700148 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:05.700620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.700430 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "kube-api-access-jhr8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:05.700620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.700496 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:54:05.700620 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.700550 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo" pod "e664a08e-94c9-442b-b835-81ee0ea0908e" (UID: "e664a08e-94c9-442b-b835-81ee0ea0908e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:54:05.739274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739232 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739274 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739270 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739283 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhr8k\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-kube-api-access-jhr8k\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739298 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e664a08e-94c9-442b-b835-81ee0ea0908e-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739310 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739321 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739334 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739347 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e664a08e-94c9-442b-b835-81ee0ea0908e-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:05.739508 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:05.739359 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e664a08e-94c9-442b-b835-81ee0ea0908e-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:06.437989 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.437951 4992 generic.go:356] "Generic (PLEG): container finished" podID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerID="ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be" exitCode=0 Nov 03 01:54:06.438173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.438001 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" Nov 03 01:54:06.438173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.438020 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" event={"ID":"e664a08e-94c9-442b-b835-81ee0ea0908e","Type":"ContainerDied","Data":"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be"} Nov 03 01:54:06.438173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.438076 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg" event={"ID":"e664a08e-94c9-442b-b835-81ee0ea0908e","Type":"ContainerDied","Data":"b0ee68e6ddafe9893109714deda0fbeb5d23972b4661773fbbc20ad5440fd503"} Nov 03 01:54:06.438173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.438101 4992 scope.go:122] "RemoveContainer" containerID="ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be" Nov 03 01:54:06.449471 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.449451 4992 scope.go:122] "RemoveContainer" containerID="ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be" Nov 03 01:54:06.449828 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:54:06.449809 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be\": container with ID starting with ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be not found: ID does not exist" containerID="ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be" Nov 03 01:54:06.449873 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.449837 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be"} err="failed to get container status \"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be\": rpc error: code = NotFound desc = could not find container \"ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be\": container with ID starting with ad5a71b2d991187d8c5627f5e6f68e9292f379f92c79bc98aa30607896e935be not found: ID does not exist" Nov 03 01:54:06.631667 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.631632 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:54:06.683466 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:06.683430 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-ct4xg"] Nov 03 01:54:08.596244 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:08.596207 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" path="/var/lib/kubelet/pods/e664a08e-94c9-442b-b835-81ee0ea0908e/volumes" Nov 03 01:54:18.193791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.193721 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:54:18.194330 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.194029 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" podUID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" containerName="maas-api" containerID="cri-o://cba122d54a1557d6bbbf4fb3f375186a518db374eab5fb4e769608e1c9a8cca5" gracePeriod=30 Nov 03 01:54:18.548901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.548506 4992 generic.go:356] "Generic (PLEG): container finished" podID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" containerID="cba122d54a1557d6bbbf4fb3f375186a518db374eab5fb4e769608e1c9a8cca5" exitCode=0 Nov 03 01:54:18.548901 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.548762 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" event={"ID":"ca74b3a0-252b-44ce-8833-98d4519fcbc9","Type":"ContainerDied","Data":"cba122d54a1557d6bbbf4fb3f375186a518db374eab5fb4e769608e1c9a8cca5"} Nov 03 01:54:18.603881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.603851 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:54:18.748087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.748051 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") pod \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " Nov 03 01:54:18.748434 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.748411 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\") pod \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " Nov 03 01:54:18.748685 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.748665 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\" (UniqueName: \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\") pod \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " Nov 03 01:54:18.748830 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.748771 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\") pod \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\" (UID: \"ca74b3a0-252b-44ce-8833-98d4519fcbc9\") " Nov 03 01:54:18.748932 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.748905 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca" pod "ca74b3a0-252b-44ce-8833-98d4519fcbc9" (UID: "ca74b3a0-252b-44ce-8833-98d4519fcbc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:54:18.749380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.749355 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca74b3a0-252b-44ce-8833-98d4519fcbc9-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:18.750965 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.750929 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls" pod "ca74b3a0-252b-44ce-8833-98d4519fcbc9" (UID: "ca74b3a0-252b-44ce-8833-98d4519fcbc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:54:18.751271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.751247 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs" pod "ca74b3a0-252b-44ce-8833-98d4519fcbc9" (UID: "ca74b3a0-252b-44ce-8833-98d4519fcbc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:54:18.751353 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.751282 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k" pod "ca74b3a0-252b-44ce-8833-98d4519fcbc9" (UID: "ca74b3a0-252b-44ce-8833-98d4519fcbc9"). InnerVolumeSpecName "kube-api-access-m4j4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:54:18.850371 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.850270 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j4k\" (UniqueName: \"kubernetes.io/projected/ca74b3a0-252b-44ce-8833-98d4519fcbc9-kube-api-access-m4j4k\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:18.850371 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.850308 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:18.850371 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:18.850320 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca74b3a0-252b-44ce-8833-98d4519fcbc9-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:54:19.561352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:19.561313 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" event={"ID":"ca74b3a0-252b-44ce-8833-98d4519fcbc9","Type":"ContainerDied","Data":"a3ea0f9ee86839fb578f21701f58473295992e0769e5b1e039cc931829d9ab84"} Nov 03 01:54:19.561352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:19.561341 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76" Nov 03 01:54:19.562022 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:19.561385 4992 scope.go:122] "RemoveContainer" containerID="cba122d54a1557d6bbbf4fb3f375186a518db374eab5fb4e769608e1c9a8cca5" Nov 03 01:54:19.734121 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:19.734083 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:54:19.790583 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:19.790519 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-86e679-6b9bd98b9b-b2k76"] Nov 03 01:54:20.594944 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:54:20.594906 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" path="/var/lib/kubelet/pods/ca74b3a0-252b-44ce-8833-98d4519fcbc9/volumes" Nov 03 01:54:32.151202 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:54:32.151162 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:55:02.152066 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:02.152026 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:55:06.019089 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.019052 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:55:06.019523 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.019277 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" podUID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" containerName="maas-api" containerID="cri-o://d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7" gracePeriod=30 Nov 03 01:55:06.390961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.390927 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:55:06.438211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.438172 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\") pod \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " Nov 03 01:55:06.438410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.438229 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\") pod \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " Nov 03 01:55:06.438410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.438336 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\") pod \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " Nov 03 01:55:06.438410 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.438398 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\") pod \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\" (UID: \"4bf47964-5aeb-4fa2-a73f-1a353b3885a2\") " Nov 03 01:55:06.438739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.438711 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca" pod "4bf47964-5aeb-4fa2-a73f-1a353b3885a2" (UID: "4bf47964-5aeb-4fa2-a73f-1a353b3885a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:06.439062 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.439041 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:06.440454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.440423 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls" pod "4bf47964-5aeb-4fa2-a73f-1a353b3885a2" (UID: "4bf47964-5aeb-4fa2-a73f-1a353b3885a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:06.440547 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.440453 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs" pod "4bf47964-5aeb-4fa2-a73f-1a353b3885a2" (UID: "4bf47964-5aeb-4fa2-a73f-1a353b3885a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:55:06.440547 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.440486 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7" pod "4bf47964-5aeb-4fa2-a73f-1a353b3885a2" (UID: "4bf47964-5aeb-4fa2-a73f-1a353b3885a2"). InnerVolumeSpecName "kube-api-access-2hfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:06.540665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.540626 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:06.540665 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.540661 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-kube-api-access-2hfc7\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:06.540916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.540675 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf47964-5aeb-4fa2-a73f-1a353b3885a2-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:06.909285 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.909240 4992 generic.go:356] "Generic (PLEG): container finished" podID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" containerID="d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7" exitCode=0 Nov 03 01:55:06.909285 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.909286 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" Nov 03 01:55:06.909530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.909285 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" event={"ID":"4bf47964-5aeb-4fa2-a73f-1a353b3885a2","Type":"ContainerDied","Data":"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7"} Nov 03 01:55:06.909530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.909450 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr" event={"ID":"4bf47964-5aeb-4fa2-a73f-1a353b3885a2","Type":"ContainerDied","Data":"c44f0dd1c83edab0c9e0dfa5d5bd70f27ee6e79fe529d7cdea6a1752b03cc388"} Nov 03 01:55:06.909530 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.909477 4992 scope.go:122] "RemoveContainer" containerID="d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7" Nov 03 01:55:06.918991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.918955 4992 scope.go:122] "RemoveContainer" containerID="d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7" Nov 03 01:55:06.919276 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:06.919252 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7\": container with ID starting with d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7 not found: ID does not exist" containerID="d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7" Nov 03 01:55:06.919324 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:06.919290 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7"} err="failed to get container status \"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7\": rpc error: code = NotFound desc = could not find container \"d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7\": container with ID starting with d2b084aad32c6ba0d26250c39c5012e2eff9b540efbeb613d7085a29ec192ac7 not found: ID does not exist" Nov 03 01:55:07.100570 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:07.100525 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:55:07.110663 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:07.110632 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e0c1fb-7fdbb7c44b-m4wpr"] Nov 03 01:55:08.595101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:08.595062 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" path="/var/lib/kubelet/pods/4bf47964-5aeb-4fa2-a73f-1a353b3885a2/volumes" Nov 03 01:55:15.720398 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:15.720353 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:55:15.720950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:15.720578 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" containerID="cri-o://2306cd1b57099d0c2d263befb4e204a9f2dbec6109bf88eb229a9ed2a1cca29e" gracePeriod=30 Nov 03 01:55:21.001526 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.001444 4992 generic.go:356] "Generic (PLEG): container finished" podID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerID="2306cd1b57099d0c2d263befb4e204a9f2dbec6109bf88eb229a9ed2a1cca29e" exitCode=0 Nov 03 01:55:21.002129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.001523 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerDied","Data":"2306cd1b57099d0c2d263befb4e204a9f2dbec6109bf88eb229a9ed2a1cca29e"} Nov 03 01:55:21.002129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.001606 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" event={"ID":"299bbb0f-8ef7-4707-8cc8-be151d796a9b","Type":"ContainerDied","Data":"8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49"} Nov 03 01:55:21.002129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.001625 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d00b38ca8a8f518b9030dab4ff6c1c0681d3bee641c4674aba2003d3b1e6f49" Nov 03 01:55:21.002129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.001637 4992 scope.go:122] "RemoveContainer" containerID="f1536425119e9972e312c2f0629cb0660893187adbbc3875f8d0a61aecc61876" Nov 03 01:55:21.002129 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.002084 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:55:21.092641 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092603 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092641 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092651 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092697 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092727 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092768 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092824 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.092910 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092908 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.093155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.092953 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:21.093155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093011 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.093155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093071 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\") pod \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\" (UID: \"299bbb0f-8ef7-4707-8cc8-be151d796a9b\") " Nov 03 01:55:21.093155 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093100 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:21.093352 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093271 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:21.093540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093508 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:21.093781 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093737 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.093909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093784 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.093909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093798 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.093909 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093813 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.094091 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.093971 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:21.095133 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.095105 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:21.095232 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.095179 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:55:21.095232 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.095217 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:21.095332 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.095267 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl" pod "299bbb0f-8ef7-4707-8cc8-be151d796a9b" (UID: "299bbb0f-8ef7-4707-8cc8-be151d796a9b"). InnerVolumeSpecName "kube-api-access-9pzpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:21.195505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.195464 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.195505 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.195506 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.195715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.195522 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzpl\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-kube-api-access-9pzpl\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.195715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.195538 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:21.195715 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:21.195552 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/299bbb0f-8ef7-4707-8cc8-be151d796a9b-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:22.008754 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:22.008720 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2" Nov 03 01:55:22.325762 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:22.321984 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:55:22.332050 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:22.331998 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e0c1fb-openshift-default-6dd744894c-fswl2"] Nov 03 01:55:22.595463 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:22.595384 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" path="/var/lib/kubelet/pods/299bbb0f-8ef7-4707-8cc8-be151d796a9b/volumes" Nov 03 01:55:32.152972 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:32.152932 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:55:32.548502 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:32.548454 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:55:32.548776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:32.548745 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" containerID="cri-o://6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505" gracePeriod=30 Nov 03 01:55:37.837920 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.837888 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:55:37.884295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884208 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884295 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884275 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884309 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884348 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884405 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884434 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884507 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884478 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884511 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:37.884740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884610 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:37.884740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884632 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884678 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\") pod \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\" (UID: \"5e1bcd02-c463-48b1-ba8f-6c47464a0feb\") " Nov 03 01:55:37.884740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884729 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:37.884941 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.884878 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:37.885108 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885071 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:37.885240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885206 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.885240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885226 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.885240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885239 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.885355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885252 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.885355 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.885267 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.886993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.886952 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:37.886993 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.886967 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "kube-api-access-rhw88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:37.887211 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.887192 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:37.887281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.887265 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo" pod "5e1bcd02-c463-48b1-ba8f-6c47464a0feb" (UID: "5e1bcd02-c463-48b1-ba8f-6c47464a0feb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:55:37.986488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.986444 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.986488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.986480 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.986488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.986490 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhw88\" (UniqueName: \"kubernetes.io/projected/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-kube-api-access-rhw88\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:37.986488 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:37.986502 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1bcd02-c463-48b1-ba8f-6c47464a0feb-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:38.141718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.141628 4992 generic.go:356] "Generic (PLEG): container finished" podID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerID="6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505" exitCode=0 Nov 03 01:55:38.141718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.141691 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerDied","Data":"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505"} Nov 03 01:55:38.141718 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.141694 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" Nov 03 01:55:38.141992 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.141740 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf" event={"ID":"5e1bcd02-c463-48b1-ba8f-6c47464a0feb","Type":"ContainerDied","Data":"ff842564f8870a7a92ddb1c6dc842a1f1c20415c62e96638df3b8fe386ab559c"} Nov 03 01:55:38.141992 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.141780 4992 scope.go:122] "RemoveContainer" containerID="6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505" Nov 03 01:55:38.156852 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.156820 4992 scope.go:122] "RemoveContainer" containerID="766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735" Nov 03 01:55:38.166380 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.166360 4992 scope.go:122] "RemoveContainer" containerID="6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505" Nov 03 01:55:38.166699 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:38.166676 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505\": container with ID starting with 6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505 not found: ID does not exist" containerID="6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505" Nov 03 01:55:38.166768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.166710 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505"} err="failed to get container status \"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505\": rpc error: code = NotFound desc = could not find container \"6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505\": container with ID starting with 6fa4ad017aa66c251dabbc2963ccc6ea0c79e71b441dd34dc98dbda7d934c505 not found: ID does not exist" Nov 03 01:55:38.166768 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.166732 4992 scope.go:122] "RemoveContainer" containerID="766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735" Nov 03 01:55:38.167041 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:38.167018 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735\": container with ID starting with 766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735 not found: ID does not exist" containerID="766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735" Nov 03 01:55:38.167081 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.167049 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735"} err="failed to get container status \"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735\": rpc error: code = NotFound desc = could not find container \"766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735\": container with ID starting with 766be61c334b811c3a14426f139539fb0c0c68180a226118e30b05e063a22735 not found: ID does not exist" Nov 03 01:55:38.315051 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.315016 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:55:38.320004 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.319975 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-71746a-openshift-default-bfc5f6d57-bksvf"] Nov 03 01:55:38.603874 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:38.603831 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" path="/var/lib/kubelet/pods/5e1bcd02-c463-48b1-ba8f-6c47464a0feb/volumes" Nov 03 01:55:52.375311 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:52.375276 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:55:52.375903 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:52.375497 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" containerID="cri-o://d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0" gracePeriod=30 Nov 03 01:55:54.854092 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:54.853989 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:55:54.854614 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:54.854274 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" containerID="cri-o://9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c" gracePeriod=30 Nov 03 01:55:57.651699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.651629 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:55:57.744776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.744742 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.744981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.744806 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.744981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.744849 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.744981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.744879 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.744996 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745064 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745152 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745106 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745145 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:57.745453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745167 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745181 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:57.745453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745251 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\") pod \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\" (UID: \"ca36178d-32d0-417f-a2f8-3a775a3d5fe9\") " Nov 03 01:55:57.745453 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745416 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745593 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.745729 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.746003 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.746031 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.746046 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.746060 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.746271 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.746075 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.747382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.747356 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:55:57.747485 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.747420 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "kube-api-access-tzrxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:57.747485 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.747463 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:55:57.747776 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.747758 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo" pod "ca36178d-32d0-417f-a2f8-3a775a3d5fe9" (UID: "ca36178d-32d0-417f-a2f8-3a775a3d5fe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:55:57.847258 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.847214 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrxz\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-kube-api-access-tzrxz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.847258 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.847252 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.847258 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.847262 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:57.847485 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:57.847270 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36178d-32d0-417f-a2f8-3a775a3d5fe9-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:55:58.292885 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.292845 4992 generic.go:356] "Generic (PLEG): container finished" podID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerID="d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0" exitCode=0 Nov 03 01:55:58.293087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.292925 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" Nov 03 01:55:58.293087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.292925 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerDied","Data":"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0"} Nov 03 01:55:58.293087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.293026 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc" event={"ID":"ca36178d-32d0-417f-a2f8-3a775a3d5fe9","Type":"ContainerDied","Data":"aa3588ae05720b7824159740adafad81b914e408a517a3a6807a7c3234838c93"} Nov 03 01:55:58.293087 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.293055 4992 scope.go:122] "RemoveContainer" containerID="d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0" Nov 03 01:55:58.303547 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.303525 4992 scope.go:122] "RemoveContainer" containerID="e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9" Nov 03 01:55:58.313777 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.313756 4992 scope.go:122] "RemoveContainer" containerID="d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0" Nov 03 01:55:58.314155 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:58.314122 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0\": container with ID starting with d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0 not found: ID does not exist" containerID="d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0" Nov 03 01:55:58.314262 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.314165 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0"} err="failed to get container status \"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0\": rpc error: code = NotFound desc = could not find container \"d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0\": container with ID starting with d602943bcbd4be4eb943308754f4102f1489c243ac92febc91f862b72df6c1c0 not found: ID does not exist" Nov 03 01:55:58.314262 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.314193 4992 scope.go:122] "RemoveContainer" containerID="e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9" Nov 03 01:55:58.314588 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:55:58.314548 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9\": container with ID starting with e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9 not found: ID does not exist" containerID="e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9" Nov 03 01:55:58.314687 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.314597 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9"} err="failed to get container status \"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9\": rpc error: code = NotFound desc = could not find container \"e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9\": container with ID starting with e01df789d7e13f128c85500d159effc51d0bfaba460f7e6150ab854ffcc6b5e9 not found: ID does not exist" Nov 03 01:55:58.470601 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.470547 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:55:58.529596 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.523615 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b37ba5-openshift-default-7bdbff4d45-m7bsc"] Nov 03 01:55:58.595347 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:55:58.595268 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" path="/var/lib/kubelet/pods/ca36178d-32d0-417f-a2f8-3a775a3d5fe9/volumes" Nov 03 01:56:00.134437 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.134376 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:56:00.177633 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177597 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.177840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177642 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.177840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177669 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.177840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177696 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.177840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177757 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.177840 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177805 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.178128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177858 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.178128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177908 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.178128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177934 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\") pod \"8b5dde6d-0680-4998-892a-ecd7ef811939\" (UID: \"8b5dde6d-0680-4998-892a-ecd7ef811939\") " Nov 03 01:56:00.178128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.177967 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:56:00.178128 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178090 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:56:00.178379 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178244 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:56:00.178379 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178273 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:56:00.178379 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178367 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:56:00.178538 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178516 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.178623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178544 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.178623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178585 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.178623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178600 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.178623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.178614 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b5dde6d-0680-4998-892a-ecd7ef811939-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.180106 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.180086 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:56:00.180218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.180199 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:00.180396 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.180378 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "kube-api-access-r848b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:56:00.180437 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.180382 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy" pod "8b5dde6d-0680-4998-892a-ecd7ef811939" (UID: "8b5dde6d-0680-4998-892a-ecd7ef811939"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:56:00.280194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.280157 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.280194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.280186 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.280194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.280198 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r848b\" (UniqueName: \"kubernetes.io/projected/8b5dde6d-0680-4998-892a-ecd7ef811939-kube-api-access-r848b\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.280454 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.280210 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b5dde6d-0680-4998-892a-ecd7ef811939-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:56:00.308520 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.308475 4992 generic.go:356] "Generic (PLEG): container finished" podID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerID="9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c" exitCode=0 Nov 03 01:56:00.308756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.308588 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" Nov 03 01:56:00.308756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.308601 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerDied","Data":"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c"} Nov 03 01:56:00.308756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.308676 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck" event={"ID":"8b5dde6d-0680-4998-892a-ecd7ef811939","Type":"ContainerDied","Data":"0dee25fd531a68cc0c7feb039523ddb206c814d74d6ce236c1ce2314d975dcd4"} Nov 03 01:56:00.308756 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.308708 4992 scope.go:122] "RemoveContainer" containerID="9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c" Nov 03 01:56:00.322494 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.322467 4992 scope.go:122] "RemoveContainer" containerID="7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e" Nov 03 01:56:00.334218 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.334192 4992 scope.go:122] "RemoveContainer" containerID="9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c" Nov 03 01:56:00.334672 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:56:00.334591 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c\": container with ID starting with 9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c not found: ID does not exist" containerID="9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c" Nov 03 01:56:00.334672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.334630 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c"} err="failed to get container status \"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c\": rpc error: code = NotFound desc = could not find container \"9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c\": container with ID starting with 9a808af467fc4228533d52644149249bab226ef6b9ee48a59b9c282feee6c36c not found: ID does not exist" Nov 03 01:56:00.334672 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.334649 4992 scope.go:122] "RemoveContainer" containerID="7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e" Nov 03 01:56:00.335008 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:56:00.334967 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e\": container with ID starting with 7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e not found: ID does not exist" containerID="7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e" Nov 03 01:56:00.335050 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.335014 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e"} err="failed to get container status \"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e\": rpc error: code = NotFound desc = could not find container \"7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e\": container with ID starting with 7ef78118e441aae7502251818e4292751a14fe217ba621f0fc0335666554fc7e not found: ID does not exist" Nov 03 01:56:00.484432 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.484389 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:56:00.488281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.488254 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-s59ck"] Nov 03 01:56:00.595658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.595616 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" path="/var/lib/kubelet/pods/8b5dde6d-0680-4998-892a-ecd7ef811939/volumes" Nov 03 01:56:00.971399 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:00.971370 4992 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" podUID="f592be42-f99a-48e7-adb0-314512784195" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/healthz\": dial tcp 10.132.0.43:8081: connect: connection refused" Nov 03 01:56:01.317545 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:01.317463 4992 generic.go:356] "Generic (PLEG): container finished" podID="f592be42-f99a-48e7-adb0-314512784195" containerID="2643013ebd7f0160dafe7b7b8e8747cbc078d38172d3f92657620ae647be53c5" exitCode=1 Nov 03 01:56:01.317545 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:01.317523 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerDied","Data":"2643013ebd7f0160dafe7b7b8e8747cbc078d38172d3f92657620ae647be53c5"} Nov 03 01:56:01.318130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:01.317583 4992 scope.go:122] "RemoveContainer" containerID="d7a6c1215a9b6407218b508a94b18a12b053bf8f3b45ec5fc1e2481c9a3aba32" Nov 03 01:56:01.318130 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:01.318041 4992 scope.go:122] "RemoveContainer" containerID="2643013ebd7f0160dafe7b7b8e8747cbc078d38172d3f92657620ae647be53c5" Nov 03 01:56:01.318854 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:01.318525 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:56:02.150997 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:56:02.150943 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:56:03.334290 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:03.334249 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" event={"ID":"f592be42-f99a-48e7-adb0-314512784195","Type":"ContainerStarted","Data":"78cc80cdec7af5943008f3f1e5450cc68124515dffa17820df634a9c94a1bb92"} Nov 03 01:56:03.334822 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:03.334342 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:56:32.156211 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:56:32.156169 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:56:34.347015 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:34.346979 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-d6v2s" Nov 03 01:56:45.159468 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:56:45.159435 4992 scope.go:122] "RemoveContainer" containerID="2306cd1b57099d0c2d263befb4e204a9f2dbec6109bf88eb229a9ed2a1cca29e" Nov 03 01:57:02.152230 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:57:02.152187 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:57:03.352113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:03.352073 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:57:03.352707 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:03.352339 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" containerID="cri-o://4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55" gracePeriod=30 Nov 03 01:57:04.778931 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:04.778893 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:57:04.779332 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:04.778949 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:57:08.629219 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.629157 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:57:08.718489 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718449 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718712 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718610 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718712 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718649 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718712 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718713 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718734 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718768 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718802 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718824 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718842 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.718891 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.718881 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\") pod \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\" (UID: \"fecd72e5-f59b-4ca5-9810-2bfd305208eb\") " Nov 03 01:57:08.719197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719006 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:08.719197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719069 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:08.719197 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:08.719937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719546 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.719937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719585 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.719937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719599 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.719937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719617 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.719937 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.719885 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:08.721229 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.721203 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:08.721310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.721229 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "kube-api-access-xjnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:08.721310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.721240 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:57:08.721310 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.721254 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token" pod "fecd72e5-f59b-4ca5-9810-2bfd305208eb" (UID: "fecd72e5-f59b-4ca5-9810-2bfd305208eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:08.761950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.761912 4992 generic.go:356] "Generic (PLEG): container finished" podID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerID="4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55" exitCode=0 Nov 03 01:57:08.762145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.761955 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" Nov 03 01:57:08.762145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.761995 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" event={"ID":"fecd72e5-f59b-4ca5-9810-2bfd305208eb","Type":"ContainerDied","Data":"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55"} Nov 03 01:57:08.762145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.762056 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr" event={"ID":"fecd72e5-f59b-4ca5-9810-2bfd305208eb","Type":"ContainerDied","Data":"d145c88eba182e77b60a560fa6d880bf14bf0d0c4f08fd324888a04cc45bf409"} Nov 03 01:57:08.762145 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.762088 4992 scope.go:122] "RemoveContainer" containerID="4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55" Nov 03 01:57:08.772724 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.772707 4992 scope.go:122] "RemoveContainer" containerID="4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55" Nov 03 01:57:08.773053 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:57:08.773031 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55\": container with ID starting with 4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55 not found: ID does not exist" containerID="4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55" Nov 03 01:57:08.773098 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.773063 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55"} err="failed to get container status \"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55\": rpc error: code = NotFound desc = could not find container \"4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55\": container with ID starting with 4ba9ab6dd74426728472c41e8c8423390141cc9ff2a68ecc55846fa1a6dc4d55 not found: ID does not exist" Nov 03 01:57:08.820895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.820864 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.820895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.820892 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.820895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.820906 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.821212 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.820919 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecd72e5-f59b-4ca5-9810-2bfd305208eb-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.821212 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.820935 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjnhb\" (UniqueName: \"kubernetes.io/projected/fecd72e5-f59b-4ca5-9810-2bfd305208eb-kube-api-access-xjnhb\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:08.922709 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.922673 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:57:08.926106 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:08.926081 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-xl7zr"] Nov 03 01:57:10.587441 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:10.587390 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:57:10.587916 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:10.587737 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" containerID="cri-o://5e1bf043971502cad04f60bd95f1457d07fc1dfc04b742e3ec0a4e3bdb1589bd" gracePeriod=30 Nov 03 01:57:10.595791 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:10.595763 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" path="/var/lib/kubelet/pods/fecd72e5-f59b-4ca5-9810-2bfd305208eb/volumes" Nov 03 01:57:13.707269 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:13.707227 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fvmnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:57:13.707693 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:13.707309 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:57:15.806860 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:15.806821 4992 generic.go:356] "Generic (PLEG): container finished" podID="15749875-6283-453a-a9d8-5c79ba723a83" containerID="5e1bf043971502cad04f60bd95f1457d07fc1dfc04b742e3ec0a4e3bdb1589bd" exitCode=0 Nov 03 01:57:15.807312 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:15.806895 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" event={"ID":"15749875-6283-453a-a9d8-5c79ba723a83","Type":"ContainerDied","Data":"5e1bf043971502cad04f60bd95f1457d07fc1dfc04b742e3ec0a4e3bdb1589bd"} Nov 03 01:57:15.860739 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:15.860665 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:57:16.004107 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004069 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004107 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004111 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004155 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004267 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004313 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004368 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004363 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004399 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:16.004604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004449 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004500 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004604 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004538 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\") pod \"15749875-6283-453a-a9d8-5c79ba723a83\" (UID: \"15749875-6283-453a-a9d8-5c79ba723a83\") " Nov 03 01:57:16.004818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004638 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:16.004818 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004680 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:16.004991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004941 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:16.004991 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.004961 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:16.005101 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.005089 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.005159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.005111 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.005159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.005125 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.005159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.005141 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.005159 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.005156 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15749875-6283-453a-a9d8-5c79ba723a83-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.006674 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.006637 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "kube-api-access-sjscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:16.006790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.006673 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:57:16.006942 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.006917 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:16.007151 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.007131 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy" pod "15749875-6283-453a-a9d8-5c79ba723a83" (UID: "15749875-6283-453a-a9d8-5c79ba723a83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:16.106796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.106744 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.106796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.106788 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15749875-6283-453a-a9d8-5c79ba723a83-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.106796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.106800 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjscm\" (UniqueName: \"kubernetes.io/projected/15749875-6283-453a-a9d8-5c79ba723a83-kube-api-access-sjscm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.106796 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.106809 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15749875-6283-453a-a9d8-5c79ba723a83-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:16.814162 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.814132 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" Nov 03 01:57:16.814609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.814133 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb" event={"ID":"15749875-6283-453a-a9d8-5c79ba723a83","Type":"ContainerDied","Data":"13a523b23872d249daf0fc4a1cde5d421cf1a93037d43c57b9e8450afa7e639c"} Nov 03 01:57:16.814609 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:16.814275 4992 scope.go:122] "RemoveContainer" containerID="5e1bf043971502cad04f60bd95f1457d07fc1dfc04b742e3ec0a4e3bdb1589bd" Nov 03 01:57:17.036273 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:17.036232 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:57:17.087947 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:17.087906 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fvmnb"] Nov 03 01:57:18.594914 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:18.594878 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15749875-6283-453a-a9d8-5c79ba723a83" path="/var/lib/kubelet/pods/15749875-6283-453a-a9d8-5c79ba723a83/volumes" Nov 03 01:57:25.710060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:25.709978 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:57:25.710491 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:25.710242 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" containerID="cri-o://a71d3e80649ac339c0ddf2fcfe0ff7df99d65aa0c6c7c4e08cf08c814dd62f7a" gracePeriod=30 Nov 03 01:57:28.229670 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:28.229626 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-8xwk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:57:28.230095 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:28.229695 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:57:30.920144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:30.919917 4992 generic.go:356] "Generic (PLEG): container finished" podID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerID="a71d3e80649ac339c0ddf2fcfe0ff7df99d65aa0c6c7c4e08cf08c814dd62f7a" exitCode=0 Nov 03 01:57:30.920144 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:30.920104 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" event={"ID":"6a5c4d0d-59b9-4e15-8d50-f14e853846a8","Type":"ContainerDied","Data":"a71d3e80649ac339c0ddf2fcfe0ff7df99d65aa0c6c7c4e08cf08c814dd62f7a"} Nov 03 01:57:30.974474 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:30.974447 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:57:31.080251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080161 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080255 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080275 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080307 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080425 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080395 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080431 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080490 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080678 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080436 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:31.080828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080726 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:31.080828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080743 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:57:31.080828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080740 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:31.080961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080847 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.080961 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.080910 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\") pod \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\" (UID: \"6a5c4d0d-59b9-4e15-8d50-f14e853846a8\") " Nov 03 01:57:31.081221 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081194 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:31.081493 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081473 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.081598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081498 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.081598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081517 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.081598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081533 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.081598 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.081548 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.082786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.082755 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:57:31.082786 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.082777 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:31.082907 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.082776 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:57:31.083084 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.083061 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw" pod "6a5c4d0d-59b9-4e15-8d50-f14e853846a8" (UID: "6a5c4d0d-59b9-4e15-8d50-f14e853846a8"). InnerVolumeSpecName "kube-api-access-cfzgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:57:31.182828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.182786 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.182828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.182823 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzgw\" (UniqueName: \"kubernetes.io/projected/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-kube-api-access-cfzgw\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.182828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.182834 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.183149 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.182843 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a5c4d0d-59b9-4e15-8d50-f14e853846a8-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 01:57:31.926774 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.926742 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" Nov 03 01:57:31.927246 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.926744 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6" event={"ID":"6a5c4d0d-59b9-4e15-8d50-f14e853846a8","Type":"ContainerDied","Data":"b2e8c521903d03b0cf8bfbac2770004da81bbca109b55a56d52d37b99869d8ff"} Nov 03 01:57:31.927246 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:31.926884 4992 scope.go:122] "RemoveContainer" containerID="a71d3e80649ac339c0ddf2fcfe0ff7df99d65aa0c6c7c4e08cf08c814dd62f7a" Nov 03 01:57:32.149244 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:32.149207 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:57:32.151683 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:57:32.151646 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:57:32.200729 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:32.200652 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-8xwk6"] Nov 03 01:57:32.595525 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:32.595446 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" path="/var/lib/kubelet/pods/6a5c4d0d-59b9-4e15-8d50-f14e853846a8/volumes" Nov 03 01:57:34.109336 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109299 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109829 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109843 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109853 4992 cpu_manager.go:414] "containerMap: removing container" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109858 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109870 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109875 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109884 4992 cpu_manager.go:414] "containerMap: removing container" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109889 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109896 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:34.109895 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109900 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109916 4992 cpu_manager.go:414] "containerMap: removing container" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109920 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109926 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109931 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109947 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109952 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109962 4992 cpu_manager.go:414] "containerMap: removing container" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109967 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109977 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.109982 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110055 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca74b3a0-252b-44ce-8833-98d4519fcbc9" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110064 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110069 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110074 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110080 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="15749875-6283-453a-a9d8-5c79ba723a83" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110086 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf47964-5aeb-4fa2-a73f-1a353b3885a2" containerName="maas-api" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110092 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110099 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110103 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110108 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5c4d0d-59b9-4e15-8d50-f14e853846a8" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110113 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110118 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="fecd72e5-f59b-4ca5-9810-2bfd305208eb" containerName="istio-proxy" Nov 03 01:57:34.110365 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.110123 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e664a08e-94c9-442b-b835-81ee0ea0908e" containerName="istio-proxy" Nov 03 01:57:34.114972 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.114935 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.169262 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.169227 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a00717-openshift-default-dockercfg-p6gmp\"" Nov 03 01:57:34.217939 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.217901 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218114 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.217954 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q8d\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218114 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218093 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218251 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218145 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218319 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218290 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218384 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218362 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218451 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218434 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218506 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218480 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.218585 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.218533 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.225052 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.225021 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 01:57:34.319969 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.319918 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320175 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320001 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320175 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320149 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320191 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320297 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320219 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320407 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320375 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320460 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320398 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320512 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320467 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q8d\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320600 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320551 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320600 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320583 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320725 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320633 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320881 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320855 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.320969 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.320951 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.321161 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.321136 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.322732 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.322702 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.323113 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.323094 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.383281 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.383210 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.383501 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.383477 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q8d\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\") pod \"e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.429067 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.429030 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:34.857692 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:57:34.857661 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306a0bde_f8a0_4227_8a3f_ca9411d9ba1e.slice/crio-9ccaa95a2c0e955a982f89dcbbb0e58bdb74e58fdf9f65a1d1a1c708d82f1b2b WatchSource:0}: Error finding container 9ccaa95a2c0e955a982f89dcbbb0e58bdb74e58fdf9f65a1d1a1c708d82f1b2b: Status 404 returned error can't find the container with id 9ccaa95a2c0e955a982f89dcbbb0e58bdb74e58fdf9f65a1d1a1c708d82f1b2b Nov 03 01:57:34.859397 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.859369 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 01:57:34.860240 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.860204 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:34.860348 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.860268 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:34.860348 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.860301 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:57:34.947623 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.947583 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerStarted","Data":"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6"} Nov 03 01:57:34.947790 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:34.947650 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerStarted","Data":"9ccaa95a2c0e955a982f89dcbbb0e58bdb74e58fdf9f65a1d1a1c708d82f1b2b"} Nov 03 01:57:35.080116 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:35.078350 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podStartSLOduration=1.078329666 podStartE2EDuration="1.078329666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:35.072178448 +0000 UTC m=+2390.651001035" watchObservedRunningTime="2026-11-03 01:57:35.078329666 +0000 UTC m=+2390.657152262" Nov 03 01:57:35.429220 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:35.429181 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:35.435444 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:35.435398 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:35.953329 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:35.953294 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:35.954322 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:35.954302 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:57:44.885391 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:44.885360 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:57:44.892204 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:44.892180 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 01:57:45.975699 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.975662 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 01:57:45.976194 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976180 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976195 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5dde6d-0680-4998-892a-ecd7ef811939" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976204 4992 cpu_manager.go:414] "containerMap: removing container" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976210 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1bcd02-c463-48b1-ba8f-6c47464a0feb" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976221 4992 cpu_manager.go:414] "containerMap: removing container" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976226 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976233 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:45.976237 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976238 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca36178d-32d0-417f-a2f8-3a775a3d5fe9" containerName="istio-proxy" Nov 03 01:57:45.976441 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.976319 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="299bbb0f-8ef7-4707-8cc8-be151d796a9b" containerName="istio-proxy" Nov 03 01:57:45.981036 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:45.981018 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.035316 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.035282 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 01:57:46.083651 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.083622 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f95773-serving-cert\"" Nov 03 01:57:46.162326 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.162290 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.162518 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.162494 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfd5\" (UniqueName: \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.162630 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.162532 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.162630 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.162550 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.264382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.264294 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfd5\" (UniqueName: \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.264382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.264338 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.264382 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.264358 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.264640 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.264419 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.265146 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.265120 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.266828 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.266806 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.266945 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.266927 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.327981 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.327943 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfd5\" (UniqueName: \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\") pod \"maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.593759 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.593652 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:46.977435 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:46.977412 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 01:57:47.019100 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:47.019067 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" event={"ID":"9d5ad328-db1e-434e-8519-dcd8f091e48e","Type":"ContainerStarted","Data":"93fcf763cef14963b1e83e8ccddd034de5be54c7f013483516b1effba104df39"} Nov 03 01:57:48.033849 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:48.033802 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" event={"ID":"9d5ad328-db1e-434e-8519-dcd8f091e48e","Type":"ContainerStarted","Data":"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a"} Nov 03 01:57:48.034341 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:48.034158 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:57:48.156514 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:48.156395 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" podStartSLOduration=2.311164286 podStartE2EDuration="3.156380087s" totalImagesPullingTime="845.215801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:48.15555047 +0000 UTC m=+2403.734373056" watchObservedRunningTime="2026-11-03 01:57:48.156380087 +0000 UTC m=+2403.735202695" Nov 03 01:57:54.047392 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:57:54.047360 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 01:58:02.150358 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:58:02.150313 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:58:32.151397 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:58:32.151349 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:58:46.880421 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.880382 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 01:58:46.884464 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.884439 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:46.939636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.939597 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 01:58:46.975423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.975384 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:46.975423 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.975425 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6m29\" (UniqueName: \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:46.975658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.975600 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:46.975658 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.975642 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:46.988540 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:46.988513 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a00717-serving-cert\"" Nov 03 01:58:47.076766 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.076729 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.076950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.076775 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.076950 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.076871 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.076950 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:58:47.076886 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-serving-cert: secret "maas-api-e2e-praxis-b-a00717-serving-cert" not found Nov 03 01:58:47.076950 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:58:47.076948 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls podName:42a3cc49-6241-4ba8-aaf7-3811848c699f nodeName:}" failed. No retries permitted until 2026-11-03 01:58:47.576932407 +0000 UTC m=+2463.155754971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls") pod "maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" (UID: "42a3cc49-6241-4ba8-aaf7-3811848c699f") : secret "maas-api-e2e-praxis-b-a00717-serving-cert" not found Nov 03 01:58:47.077115 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.076889 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6m29\" (UniqueName: \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.077592 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.077550 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.079360 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.079340 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.142025 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.141932 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6m29\" (UniqueName: \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.588854 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.588810 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.591385 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.591357 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a00717-7c65464895-92zt8\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:47.798060 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:47.798022 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:48.172179 ip-10-0-138-19 kubenswrapper[4992]: W1103 01:58:48.172130 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a3cc49_6241_4ba8_aaf7_3811848c699f.slice/crio-516a6158e902b6a431230dcedc075654effa563ae1e1811e4b1af796092968d3 WatchSource:0}: Error finding container 516a6158e902b6a431230dcedc075654effa563ae1e1811e4b1af796092968d3: Status 404 returned error can't find the container with id 516a6158e902b6a431230dcedc075654effa563ae1e1811e4b1af796092968d3 Nov 03 01:58:48.174418 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:48.174388 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 01:58:48.400071 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:48.400025 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" event={"ID":"42a3cc49-6241-4ba8-aaf7-3811848c699f","Type":"ContainerStarted","Data":"516a6158e902b6a431230dcedc075654effa563ae1e1811e4b1af796092968d3"} Nov 03 01:58:49.406908 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:49.406863 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" event={"ID":"42a3cc49-6241-4ba8-aaf7-3811848c699f","Type":"ContainerStarted","Data":"f46cabe976b4e128bf1ab03a4aeeb0c7c036853bc9d6998298bd1ba4f93c39ff"} Nov 03 01:58:49.407388 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:49.406940 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:58:49.532309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:49.532202 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" podStartSLOduration=2.5416263260000003 podStartE2EDuration="3.532187173s" totalImagesPullingTime="990.560847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:58:49.529250538 +0000 UTC m=+2465.108073114" watchObservedRunningTime="2026-11-03 01:58:49.532187173 +0000 UTC m=+2465.111009759" Nov 03 01:58:55.419544 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:58:55.419461 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 01:59:02.151152 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:59:02.151112 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 01:59:16.604636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.604590 4992 generic.go:356] "Generic (PLEG): container finished" podID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerID="0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6" exitCode=0 Nov 03 01:59:16.605260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.604646 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerDied","Data":"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6"} Nov 03 01:59:16.605260 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.605152 4992 scope.go:122] "RemoveContainer" containerID="0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6" Nov 03 01:59:16.608003 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.607969 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:59:16.608137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.608048 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:59:16.608137 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:16.608085 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 01:59:17.612243 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:17.612209 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerStarted","Data":"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684"} Nov 03 01:59:18.429740 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:18.429690 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:59:18.431173 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:18.431143 4992 patch_prober.go:29] interesting pod/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 01:59:18.431289 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:18.431202 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 01:59:19.430028 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:19.429984 4992 patch_prober.go:29] interesting pod/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 01:59:19.430529 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:19.430053 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 01:59:20.429420 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:20.429385 4992 patch_prober.go:29] interesting pod/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 01:59:20.429625 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:20.429440 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 01:59:20.953309 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:20.953273 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:59:21.429940 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:21.429893 4992 patch_prober.go:29] interesting pod/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 01:59:21.430308 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:21.429960 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 01:59:22.430266 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:22.430224 4992 patch_prober.go:29] interesting pod/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 03 01:59:22.430745 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:22.430293 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 03 01:59:23.434636 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:23.434602 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:59:23.435664 ip-10-0-138-19 kubenswrapper[4992]: I1103 01:59:23.435644 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 01:59:32.151933 ip-10-0-138-19 kubenswrapper[4992]: E1103 01:59:32.151887 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:00:00.167190 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.167148 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:00.171702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.171680 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.188720 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.188688 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:00.192611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.192588 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:00.192739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.192716 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.197231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.197203 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.218852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.218826 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:00.222866 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.222843 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.238237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.238213 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:00:00.260363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.260319 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:00.300028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.299991 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcvk\" (UniqueName: \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.300233 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300200 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzh9\" (UniqueName: \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.300373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300299 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qlh\" (UniqueName: \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.300434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300371 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.300523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300508 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.300594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300543 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.300747 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300722 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfh48\" (UniqueName: \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.300825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.300802 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.308257 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.308226 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:00.311516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.311491 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:00.318399 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.318367 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:00.402819 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.402773 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.403034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.402833 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.403034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.402918 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfh48\" (UniqueName: \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.403034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.402973 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.403207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403065 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcvk\" (UniqueName: \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.403207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403103 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzh9\" (UniqueName: \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.403207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403153 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qlh\" (UniqueName: \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.403207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403196 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.403812 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403736 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.403933 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403810 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.403933 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403921 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.404019 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.403971 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.466524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.466439 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfh48\" (UniqueName: \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\") pod \"maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.466524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.466439 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qlh\" (UniqueName: \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\") pod \"maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.466744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.466526 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcvk\" (UniqueName: \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\") pod \"maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.466744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.466550 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzh9\" (UniqueName: \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.485666 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.485637 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:00.509344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.509315 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:00.515591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.515525 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:00.538459 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.538197 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:00.868217 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:00:00.868182 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63c8b47_42be_4bda_8d07_36df21a05d2d.slice/crio-34d54c63202d13580db3a53ed0b46b7e9aec3042d960080ca12f1a88e6079568 WatchSource:0}: Error finding container 34d54c63202d13580db3a53ed0b46b7e9aec3042d960080ca12f1a88e6079568: Status 404 returned error can't find the container with id 34d54c63202d13580db3a53ed0b46b7e9aec3042d960080ca12f1a88e6079568 Nov 03 02:00:00.870890 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.870825 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:00.885831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.885801 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" event={"ID":"d63c8b47-42be-4bda-8d07-36df21a05d2d","Type":"ContainerStarted","Data":"34d54c63202d13580db3a53ed0b46b7e9aec3042d960080ca12f1a88e6079568"} Nov 03 02:00:00.947444 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:00:00.947413 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129b9fd5_f0c1_483a_b391_c214c014ef52.slice/crio-ff5391e952a51ca4229cef720c62bfbce63b406f01fef1d1c0b3eabca3765360 WatchSource:0}: Error finding container ff5391e952a51ca4229cef720c62bfbce63b406f01fef1d1c0b3eabca3765360: Status 404 returned error can't find the container with id ff5391e952a51ca4229cef720c62bfbce63b406f01fef1d1c0b3eabca3765360 Nov 03 02:00:00.950469 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.950443 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:00.976610 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.976580 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:00.982882 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:00:00.982856 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3e5a8b_ce96_4eb9_a1df_4926b5eed06f.slice/crio-f4cd7bfeac6245a981dd04472c1fb54fde2a9245f2040576c57d25a5379ab276 WatchSource:0}: Error finding container f4cd7bfeac6245a981dd04472c1fb54fde2a9245f2040576c57d25a5379ab276: Status 404 returned error can't find the container with id f4cd7bfeac6245a981dd04472c1fb54fde2a9245f2040576c57d25a5379ab276 Nov 03 02:00:00.995176 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:00:00.995137 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ef3c61_064e_4c42_93ce_c55c11aabb79.slice/crio-a6de462c36e2c88bde7255d1e8ec6d2e54bc7d63a8002026f23f9c53bf828350 WatchSource:0}: Error finding container a6de462c36e2c88bde7255d1e8ec6d2e54bc7d63a8002026f23f9c53bf828350: Status 404 returned error can't find the container with id a6de462c36e2c88bde7255d1e8ec6d2e54bc7d63a8002026f23f9c53bf828350 Nov 03 02:00:00.999353 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:00.998022 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:01.894364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.894325 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerStarted","Data":"8375b970338f9266d65ee5bb93020c3f39336979287423554127a5e6e14ba7b7"} Nov 03 02:00:01.894826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.894397 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerStarted","Data":"ff5391e952a51ca4229cef720c62bfbce63b406f01fef1d1c0b3eabca3765360"} Nov 03 02:00:01.896140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.896113 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerStarted","Data":"0e3a975515c71da93bf89e01b2e6ef12c5c1da10c5c628a8083ef02ac85c52d3"} Nov 03 02:00:01.896254 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.896165 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerStarted","Data":"a6de462c36e2c88bde7255d1e8ec6d2e54bc7d63a8002026f23f9c53bf828350"} Nov 03 02:00:01.897728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.897689 4992 generic.go:356] "Generic (PLEG): container finished" podID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerID="4af4b8d807264d54c930b29b2c4ec6804bb723a2270f644b2082cde1a41c9ed9" exitCode=60 Nov 03 02:00:01.897846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.897790 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" event={"ID":"d63c8b47-42be-4bda-8d07-36df21a05d2d","Type":"ContainerDied","Data":"4af4b8d807264d54c930b29b2c4ec6804bb723a2270f644b2082cde1a41c9ed9"} Nov 03 02:00:01.898001 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.897984 4992 scope.go:122] "RemoveContainer" containerID="4af4b8d807264d54c930b29b2c4ec6804bb723a2270f644b2082cde1a41c9ed9" Nov 03 02:00:01.899644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.899603 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerStarted","Data":"c118f193ea47879af6b1c3b3e26c2a10d9a11705d7b0e960c2f0e1786a34cf53"} Nov 03 02:00:01.899699 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:01.899655 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerStarted","Data":"f4cd7bfeac6245a981dd04472c1fb54fde2a9245f2040576c57d25a5379ab276"} Nov 03 02:00:02.013636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.013577 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" podStartSLOduration=2.013548169 podStartE2EDuration="2.013548169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.012482342 +0000 UTC m=+2537.591304932" watchObservedRunningTime="2026-11-03 02:00:02.013548169 +0000 UTC m=+2537.592370754" Nov 03 02:00:02.133866 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.133730 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" podStartSLOduration=2.133701342 podStartE2EDuration="2.133701342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.131309352 +0000 UTC m=+2537.710131938" watchObservedRunningTime="2026-11-03 02:00:02.133701342 +0000 UTC m=+2537.712523928" Nov 03 02:00:02.153668 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:02.153635 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:00:02.370504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.370449 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" podStartSLOduration=2.370434763 podStartE2EDuration="2.370434763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.368681832 +0000 UTC m=+2537.947504418" watchObservedRunningTime="2026-11-03 02:00:02.370434763 +0000 UTC m=+2537.949257364" Nov 03 02:00:02.910261 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.910224 4992 generic.go:356] "Generic (PLEG): container finished" podID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerID="c118f193ea47879af6b1c3b3e26c2a10d9a11705d7b0e960c2f0e1786a34cf53" exitCode=60 Nov 03 02:00:02.910734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.910313 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerDied","Data":"c118f193ea47879af6b1c3b3e26c2a10d9a11705d7b0e960c2f0e1786a34cf53"} Nov 03 02:00:02.910734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.910635 4992 scope.go:122] "RemoveContainer" containerID="c118f193ea47879af6b1c3b3e26c2a10d9a11705d7b0e960c2f0e1786a34cf53" Nov 03 02:00:02.912210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.912183 4992 generic.go:356] "Generic (PLEG): container finished" podID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerID="8375b970338f9266d65ee5bb93020c3f39336979287423554127a5e6e14ba7b7" exitCode=60 Nov 03 02:00:02.912341 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.912285 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerDied","Data":"8375b970338f9266d65ee5bb93020c3f39336979287423554127a5e6e14ba7b7"} Nov 03 02:00:02.912533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.912514 4992 scope.go:122] "RemoveContainer" containerID="8375b970338f9266d65ee5bb93020c3f39336979287423554127a5e6e14ba7b7" Nov 03 02:00:02.914319 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.914283 4992 generic.go:356] "Generic (PLEG): container finished" podID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerID="0e3a975515c71da93bf89e01b2e6ef12c5c1da10c5c628a8083ef02ac85c52d3" exitCode=60 Nov 03 02:00:02.914319 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.914307 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerDied","Data":"0e3a975515c71da93bf89e01b2e6ef12c5c1da10c5c628a8083ef02ac85c52d3"} Nov 03 02:00:02.914612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.914595 4992 scope.go:122] "RemoveContainer" containerID="0e3a975515c71da93bf89e01b2e6ef12c5c1da10c5c628a8083ef02ac85c52d3" Nov 03 02:00:02.916662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.916637 4992 generic.go:356] "Generic (PLEG): container finished" podID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerID="6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f" exitCode=60 Nov 03 02:00:02.916718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.916678 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" event={"ID":"d63c8b47-42be-4bda-8d07-36df21a05d2d","Type":"ContainerDied","Data":"6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f"} Nov 03 02:00:02.916758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.916720 4992 scope.go:122] "RemoveContainer" containerID="4af4b8d807264d54c930b29b2c4ec6804bb723a2270f644b2082cde1a41c9ed9" Nov 03 02:00:02.916881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:02.916867 4992 scope.go:122] "RemoveContainer" containerID="6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f" Nov 03 02:00:02.917075 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:02.917052 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2_odh-ai-gateway-infra(d63c8b47-42be-4bda-8d07-36df21a05d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" Nov 03 02:00:03.924165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.924134 4992 generic.go:356] "Generic (PLEG): container finished" podID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerID="a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f" exitCode=60 Nov 03 02:00:03.924592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.924213 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerDied","Data":"a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f"} Nov 03 02:00:03.924592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.924275 4992 scope.go:122] "RemoveContainer" containerID="8375b970338f9266d65ee5bb93020c3f39336979287423554127a5e6e14ba7b7" Nov 03 02:00:03.924592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.924539 4992 scope.go:122] "RemoveContainer" containerID="a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f" Nov 03 02:00:03.924890 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:03.924865 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr_odh-ai-gateway-infra(129b9fd5-f0c1-483a-b391-c214c014ef52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" Nov 03 02:00:03.926426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.926397 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerStarted","Data":"661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb"} Nov 03 02:00:03.928200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.928183 4992 scope.go:122] "RemoveContainer" containerID="6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f" Nov 03 02:00:03.928456 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:03.928428 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2_odh-ai-gateway-infra(d63c8b47-42be-4bda-8d07-36df21a05d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" Nov 03 02:00:03.929861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.929838 4992 generic.go:356] "Generic (PLEG): container finished" podID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerID="fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc" exitCode=60 Nov 03 02:00:03.929973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.929879 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerDied","Data":"fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc"} Nov 03 02:00:03.930097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.930080 4992 scope.go:122] "RemoveContainer" containerID="fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc" Nov 03 02:00:03.930308 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:03.930288 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7_odh-ai-gateway-infra(6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" Nov 03 02:00:03.936799 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:03.936777 4992 scope.go:122] "RemoveContainer" containerID="c118f193ea47879af6b1c3b3e26c2a10d9a11705d7b0e960c2f0e1786a34cf53" Nov 03 02:00:04.937190 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.937161 4992 scope.go:122] "RemoveContainer" containerID="a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f" Nov 03 02:00:04.937663 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:04.937522 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr_odh-ai-gateway-infra(129b9fd5-f0c1-483a-b391-c214c014ef52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" Nov 03 02:00:04.938861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.938833 4992 generic.go:356] "Generic (PLEG): container finished" podID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerID="661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb" exitCode=60 Nov 03 02:00:04.938985 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.938933 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerDied","Data":"661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb"} Nov 03 02:00:04.939042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.938997 4992 scope.go:122] "RemoveContainer" containerID="0e3a975515c71da93bf89e01b2e6ef12c5c1da10c5c628a8083ef02ac85c52d3" Nov 03 02:00:04.939265 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.939245 4992 scope.go:122] "RemoveContainer" containerID="661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb" Nov 03 02:00:04.939601 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:04.939573 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz_odh-ai-gateway-infra(46ef3c61-064e-4c42-93ce-c55c11aabb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" Nov 03 02:00:04.943570 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:04.943537 4992 scope.go:122] "RemoveContainer" containerID="fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc" Nov 03 02:00:04.943902 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:04.943882 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7_odh-ai-gateway-infra(6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" Nov 03 02:00:05.951125 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:05.951097 4992 scope.go:122] "RemoveContainer" containerID="661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb" Nov 03 02:00:05.951533 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:05.951384 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz_odh-ai-gateway-infra(46ef3c61-064e-4c42-93ce-c55c11aabb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" Nov 03 02:00:15.588265 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:15.588233 4992 scope.go:122] "RemoveContainer" containerID="661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb" Nov 03 02:00:15.588722 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:15.588548 4992 scope.go:122] "RemoveContainer" containerID="6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f" Nov 03 02:00:17.026776 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.026739 4992 generic.go:356] "Generic (PLEG): container finished" podID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerID="7ad77c12b2fa131ab5967a0f7eb3e029591a2b4a96a6f2bad9ee9ff1cec9ee8b" exitCode=60 Nov 03 02:00:17.027226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.026812 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerDied","Data":"7ad77c12b2fa131ab5967a0f7eb3e029591a2b4a96a6f2bad9ee9ff1cec9ee8b"} Nov 03 02:00:17.027226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.026884 4992 scope.go:122] "RemoveContainer" containerID="661eaa58a4b058c20f5087b1eb258cf2c489fca366e10e95f0be541bb39f82cb" Nov 03 02:00:17.027328 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.027272 4992 scope.go:122] "RemoveContainer" containerID="7ad77c12b2fa131ab5967a0f7eb3e029591a2b4a96a6f2bad9ee9ff1cec9ee8b" Nov 03 02:00:17.027616 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:17.027587 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz_odh-ai-gateway-infra(46ef3c61-064e-4c42-93ce-c55c11aabb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" Nov 03 02:00:17.028788 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.028766 4992 generic.go:356] "Generic (PLEG): container finished" podID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerID="7fddd4edb84aa31c03acb244d78c25965cd3812d2d89de7100c09fccae4ae558" exitCode=60 Nov 03 02:00:17.028877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.028839 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" event={"ID":"d63c8b47-42be-4bda-8d07-36df21a05d2d","Type":"ContainerDied","Data":"7fddd4edb84aa31c03acb244d78c25965cd3812d2d89de7100c09fccae4ae558"} Nov 03 02:00:17.029177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.029164 4992 scope.go:122] "RemoveContainer" containerID="7fddd4edb84aa31c03acb244d78c25965cd3812d2d89de7100c09fccae4ae558" Nov 03 02:00:17.029417 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:17.029401 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2_odh-ai-gateway-infra(d63c8b47-42be-4bda-8d07-36df21a05d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" Nov 03 02:00:17.039821 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.039799 4992 scope.go:122] "RemoveContainer" containerID="6d45fee469d26ce6c6d86f0655fdf5addaa91950826a035a47e042c02bf0f90f" Nov 03 02:00:17.588070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.588038 4992 scope.go:122] "RemoveContainer" containerID="a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f" Nov 03 02:00:17.588220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:17.588167 4992 scope.go:122] "RemoveContainer" containerID="fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc" Nov 03 02:00:18.037235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.037188 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerStarted","Data":"e70c888e03447107d8342a841323abbcf9c620b7f32da18a8a2fbe0576baa5be"} Nov 03 02:00:18.044133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.044102 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerStarted","Data":"9e446f4ff815833ff0bb914b9953fdee909aa8feff875bbef4bd4d1894350221"} Nov 03 02:00:18.164306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.164268 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:18.280952 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.280913 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:18.323686 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.323652 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:18.429100 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.429071 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:18.432300 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.432278 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\") pod \"46ef3c61-064e-4c42-93ce-c55c11aabb79\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " Nov 03 02:00:18.432400 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.432384 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\" (UniqueName: \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\") pod \"46ef3c61-064e-4c42-93ce-c55c11aabb79\" (UID: \"46ef3c61-064e-4c42-93ce-c55c11aabb79\") " Nov 03 02:00:18.432701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.432680 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca" pod "46ef3c61-064e-4c42-93ce-c55c11aabb79" (UID: "46ef3c61-064e-4c42-93ce-c55c11aabb79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:18.432945 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.432930 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ef3c61-064e-4c42-93ce-c55c11aabb79-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:18.434393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.434371 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9" pod "46ef3c61-064e-4c42-93ce-c55c11aabb79" (UID: "46ef3c61-064e-4c42-93ce-c55c11aabb79"). InnerVolumeSpecName "kube-api-access-htzh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:18.533721 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.533679 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\") pod \"d63c8b47-42be-4bda-8d07-36df21a05d2d\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " Nov 03 02:00:18.533907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.533739 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\" (UniqueName: \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\") pod \"d63c8b47-42be-4bda-8d07-36df21a05d2d\" (UID: \"d63c8b47-42be-4bda-8d07-36df21a05d2d\") " Nov 03 02:00:18.534198 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.534136 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca" pod "d63c8b47-42be-4bda-8d07-36df21a05d2d" (UID: "d63c8b47-42be-4bda-8d07-36df21a05d2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:18.534380 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.534340 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzh9\" (UniqueName: \"kubernetes.io/projected/46ef3c61-064e-4c42-93ce-c55c11aabb79-kube-api-access-htzh9\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:18.534380 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.534383 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63c8b47-42be-4bda-8d07-36df21a05d2d-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:18.536363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.536336 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk" pod "d63c8b47-42be-4bda-8d07-36df21a05d2d" (UID: "d63c8b47-42be-4bda-8d07-36df21a05d2d"). InnerVolumeSpecName "kube-api-access-cwcvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:18.636120 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:18.636088 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwcvk\" (UniqueName: \"kubernetes.io/projected/d63c8b47-42be-4bda-8d07-36df21a05d2d-kube-api-access-cwcvk\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:19.053934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.053899 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" Nov 03 02:00:19.054413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.053899 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz" event={"ID":"46ef3c61-064e-4c42-93ce-c55c11aabb79","Type":"ContainerDied","Data":"a6de462c36e2c88bde7255d1e8ec6d2e54bc7d63a8002026f23f9c53bf828350"} Nov 03 02:00:19.054413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.054034 4992 scope.go:122] "RemoveContainer" containerID="7ad77c12b2fa131ab5967a0f7eb3e029591a2b4a96a6f2bad9ee9ff1cec9ee8b" Nov 03 02:00:19.055414 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.055385 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" event={"ID":"d63c8b47-42be-4bda-8d07-36df21a05d2d","Type":"ContainerDied","Data":"34d54c63202d13580db3a53ed0b46b7e9aec3042d960080ca12f1a88e6079568"} Nov 03 02:00:19.055523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.055393 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2" Nov 03 02:00:19.058111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.058050 4992 generic.go:356] "Generic (PLEG): container finished" podID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerID="9e446f4ff815833ff0bb914b9953fdee909aa8feff875bbef4bd4d1894350221" exitCode=60 Nov 03 02:00:19.058111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.058099 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerDied","Data":"9e446f4ff815833ff0bb914b9953fdee909aa8feff875bbef4bd4d1894350221"} Nov 03 02:00:19.058723 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.058704 4992 scope.go:122] "RemoveContainer" containerID="9e446f4ff815833ff0bb914b9953fdee909aa8feff875bbef4bd4d1894350221" Nov 03 02:00:19.058963 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:19.058942 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7_odh-ai-gateway-infra(6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" Nov 03 02:00:19.060887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.060862 4992 generic.go:356] "Generic (PLEG): container finished" podID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerID="e70c888e03447107d8342a841323abbcf9c620b7f32da18a8a2fbe0576baa5be" exitCode=60 Nov 03 02:00:19.060984 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.060906 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerDied","Data":"e70c888e03447107d8342a841323abbcf9c620b7f32da18a8a2fbe0576baa5be"} Nov 03 02:00:19.061229 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.061211 4992 scope.go:122] "RemoveContainer" containerID="e70c888e03447107d8342a841323abbcf9c620b7f32da18a8a2fbe0576baa5be" Nov 03 02:00:19.061444 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:19.061425 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr_odh-ai-gateway-infra(129b9fd5-f0c1-483a-b391-c214c014ef52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" Nov 03 02:00:19.064804 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.064789 4992 scope.go:122] "RemoveContainer" containerID="7fddd4edb84aa31c03acb244d78c25965cd3812d2d89de7100c09fccae4ae558" Nov 03 02:00:19.074464 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.074445 4992 scope.go:122] "RemoveContainer" containerID="fd1160cb7c1c7dcdce16a598bd2a5909bfb0642fc78f004ebf4451d0905953cc" Nov 03 02:00:19.087648 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.087623 4992 scope.go:122] "RemoveContainer" containerID="a3884cc0597676ded0629d5851129080f45b9d36e16bdc60d0e972fd9022f90f" Nov 03 02:00:19.168550 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.168514 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:19.226287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.226248 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:19.242591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.240760 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-e2db44-29894520-gw2c2"] Nov 03 02:00:19.284319 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.284281 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:19.587006 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.586959 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:19.593058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:19.593027 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894520-487rz"] Nov 03 02:00:20.270942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.270917 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:20.273311 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.273289 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:20.357907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.357815 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\") pod \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " Nov 03 02:00:20.357907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.357877 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\" (UniqueName: \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\") pod \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\" (UID: \"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f\") " Nov 03 02:00:20.357907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.357898 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\" (UniqueName: \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\") pod \"129b9fd5-f0c1-483a-b391-c214c014ef52\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " Nov 03 02:00:20.358214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.357955 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\") pod \"129b9fd5-f0c1-483a-b391-c214c014ef52\" (UID: \"129b9fd5-f0c1-483a-b391-c214c014ef52\") " Nov 03 02:00:20.358316 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.358288 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca" pod "6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" (UID: "6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:20.358450 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.358426 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca" pod "129b9fd5-f0c1-483a-b391-c214c014ef52" (UID: "129b9fd5-f0c1-483a-b391-c214c014ef52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:00:20.360191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.360164 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh" pod "6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" (UID: "6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f"). InnerVolumeSpecName "kube-api-access-89qlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:20.360464 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.360434 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48" pod "129b9fd5-f0c1-483a-b391-c214c014ef52" (UID: "129b9fd5-f0c1-483a-b391-c214c014ef52"). InnerVolumeSpecName "kube-api-access-wfh48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:00:20.460021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.459978 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:20.460021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.460014 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qlh\" (UniqueName: \"kubernetes.io/projected/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f-kube-api-access-89qlh\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:20.460021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.460025 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfh48\" (UniqueName: \"kubernetes.io/projected/129b9fd5-f0c1-483a-b391-c214c014ef52-kube-api-access-wfh48\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:20.460258 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.460034 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/129b9fd5-f0c1-483a-b391-c214c014ef52-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:00:20.594567 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.594519 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" path="/var/lib/kubelet/pods/46ef3c61-064e-4c42-93ce-c55c11aabb79/volumes" Nov 03 02:00:20.595046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:20.595030 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" path="/var/lib/kubelet/pods/d63c8b47-42be-4bda-8d07-36df21a05d2d/volumes" Nov 03 02:00:21.078613 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.078584 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" Nov 03 02:00:21.078613 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.078591 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr" event={"ID":"129b9fd5-f0c1-483a-b391-c214c014ef52","Type":"ContainerDied","Data":"ff5391e952a51ca4229cef720c62bfbce63b406f01fef1d1c0b3eabca3765360"} Nov 03 02:00:21.078894 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.078649 4992 scope.go:122] "RemoveContainer" containerID="e70c888e03447107d8342a841323abbcf9c620b7f32da18a8a2fbe0576baa5be" Nov 03 02:00:21.080320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.080302 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" Nov 03 02:00:21.080427 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.080354 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7" event={"ID":"6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f","Type":"ContainerDied","Data":"f4cd7bfeac6245a981dd04472c1fb54fde2a9245f2040576c57d25a5379ab276"} Nov 03 02:00:21.088397 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.088376 4992 scope.go:122] "RemoveContainer" containerID="9e446f4ff815833ff0bb914b9953fdee909aa8feff875bbef4bd4d1894350221" Nov 03 02:00:21.251096 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.251062 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:21.302508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.302454 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a00717-29894520-8fxm7"] Nov 03 02:00:21.372754 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.372668 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:21.380644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:21.380608 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-706bdd-29894520-s42xr"] Nov 03 02:00:22.594255 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:22.594220 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" path="/var/lib/kubelet/pods/129b9fd5-f0c1-483a-b391-c214c014ef52/volumes" Nov 03 02:00:22.594665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:00:22.594654 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" path="/var/lib/kubelet/pods/6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f/volumes" Nov 03 02:00:32.154957 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:00:32.154912 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:01:00.568467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.568427 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 02:01:00.568970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.568649 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" podUID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" containerName="maas-api" containerID="cri-o://00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c" gracePeriod=30 Nov 03 02:01:00.867955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.867919 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 02:01:00.951594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.951326 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\") pod \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " Nov 03 02:01:00.951594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.951385 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\" (UniqueName: \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\") pod \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " Nov 03 02:01:00.951594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.951481 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\") pod \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " Nov 03 02:01:00.951594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.951600 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") pod \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\" (UID: \"0ef0bc65-ba8a-4e7f-878f-530c05131b6d\") " Nov 03 02:01:00.953171 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.952829 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca" pod "0ef0bc65-ba8a-4e7f-878f-530c05131b6d" (UID: "0ef0bc65-ba8a-4e7f-878f-530c05131b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:00.955924 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.955895 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs" pod "0ef0bc65-ba8a-4e7f-878f-530c05131b6d" (UID: "0ef0bc65-ba8a-4e7f-878f-530c05131b6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:00.960318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.960284 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls" pod "0ef0bc65-ba8a-4e7f-878f-530c05131b6d" (UID: "0ef0bc65-ba8a-4e7f-878f-530c05131b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:01:00.965854 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:00.965808 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb" pod "0ef0bc65-ba8a-4e7f-878f-530c05131b6d" (UID: "0ef0bc65-ba8a-4e7f-878f-530c05131b6d"). InnerVolumeSpecName "kube-api-access-wf6vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:01.053875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.053837 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:01.053875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.053871 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:01.054105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.053884 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf6vb\" (UniqueName: \"kubernetes.io/projected/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-kube-api-access-wf6vb\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:01.054105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.053896 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ef0bc65-ba8a-4e7f-878f-530c05131b6d-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:01.325133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.325093 4992 generic.go:356] "Generic (PLEG): container finished" podID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" containerID="00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c" exitCode=0 Nov 03 02:01:01.325333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.325149 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" Nov 03 02:01:01.325333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.325178 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" event={"ID":"0ef0bc65-ba8a-4e7f-878f-530c05131b6d","Type":"ContainerDied","Data":"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c"} Nov 03 02:01:01.325333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.325234 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l" event={"ID":"0ef0bc65-ba8a-4e7f-878f-530c05131b6d","Type":"ContainerDied","Data":"90a7a982772fc16a46dd9ce9c6bc02c879ae3cd70812c9697c5e177d655c81ad"} Nov 03 02:01:01.325333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.325259 4992 scope.go:122] "RemoveContainer" containerID="00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c" Nov 03 02:01:01.336073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.336052 4992 scope.go:122] "RemoveContainer" containerID="00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c" Nov 03 02:01:01.336351 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:01:01.336329 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c\": container with ID starting with 00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c not found: ID does not exist" containerID="00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c" Nov 03 02:01:01.336401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.336363 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c"} err="failed to get container status \"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c\": rpc error: code = NotFound desc = could not find container \"00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c\": container with ID starting with 00c5d670ec791f0298dce71831ad75e2a31f859a9291424948fde614d9ab266c not found: ID does not exist" Nov 03 02:01:01.498579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.498520 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 02:01:01.503363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:01.503325 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b177c9-dc6fc6bb5-h6c8l"] Nov 03 02:01:02.151764 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:01:02.151722 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:01:02.598028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:02.597990 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" path="/var/lib/kubelet/pods/0ef0bc65-ba8a-4e7f-878f-530c05131b6d/volumes" Nov 03 02:01:02.857012 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:02.856919 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 02:01:02.857229 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:02.857201 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" containerID="cri-o://0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192" gracePeriod=30 Nov 03 02:01:08.123737 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.123704 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 02:01:08.163960 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.163922 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.163963 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164018 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164064 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164098 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164133 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164176 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164281 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164279 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:08.164406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164381 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:08.164406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164394 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:08.164713 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164464 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\") pod \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\" (UID: \"a22fd95a-ae21-4278-bc3d-3e9e5afb1298\") " Nov 03 02:01:08.164713 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164697 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:08.164832 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164730 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:01:08.165060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164975 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.165060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.164997 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.165060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.165011 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.165060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.165025 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.165060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.165038 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.166367 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.166346 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:01:08.166647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.166623 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "kube-api-access-lnpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:08.166908 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.166884 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:01:08.166982 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.166967 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo" pod "a22fd95a-ae21-4278-bc3d-3e9e5afb1298" (UID: "a22fd95a-ae21-4278-bc3d-3e9e5afb1298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:01:08.266018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.265972 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.266018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.266009 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.266018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.266019 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.266018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.266029 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnpw2\" (UniqueName: \"kubernetes.io/projected/a22fd95a-ae21-4278-bc3d-3e9e5afb1298-kube-api-access-lnpw2\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:01:08.371731 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.371698 4992 generic.go:356] "Generic (PLEG): container finished" podID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerID="0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192" exitCode=0 Nov 03 02:01:08.371917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.371744 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" Nov 03 02:01:08.371917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.371784 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerDied","Data":"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192"} Nov 03 02:01:08.371917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.371873 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r" event={"ID":"a22fd95a-ae21-4278-bc3d-3e9e5afb1298","Type":"ContainerDied","Data":"6dc01b296c8aa85ab8f0a9fc0c332fcbe5443d0dfdcefd81f9276b0cd509fda5"} Nov 03 02:01:08.371917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.371903 4992 scope.go:122] "RemoveContainer" containerID="0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192" Nov 03 02:01:08.385779 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.385702 4992 scope.go:122] "RemoveContainer" containerID="e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10" Nov 03 02:01:08.406524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.406491 4992 scope.go:122] "RemoveContainer" containerID="0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192" Nov 03 02:01:08.406926 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:01:08.406899 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192\": container with ID starting with 0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192 not found: ID does not exist" containerID="0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192" Nov 03 02:01:08.407033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.406930 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192"} err="failed to get container status \"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192\": rpc error: code = NotFound desc = could not find container \"0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192\": container with ID starting with 0bb272934c8ccb52fbd558ab999c48e1bcfda7e8fc69c3e507aa8127df0fb192 not found: ID does not exist" Nov 03 02:01:08.407033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.406950 4992 scope.go:122] "RemoveContainer" containerID="e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10" Nov 03 02:01:08.407282 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:01:08.407256 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10\": container with ID starting with e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10 not found: ID does not exist" containerID="e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10" Nov 03 02:01:08.407341 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.407294 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10"} err="failed to get container status \"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10\": rpc error: code = NotFound desc = could not find container \"e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10\": container with ID starting with e10ad3b5f118154fd16f714ae42331e6743b8b21cc62b0b48c3a8868f86d2b10 not found: ID does not exist" Nov 03 02:01:08.550643 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.547582 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 02:01:08.554881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.554843 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b177c9-openshift-default-55d7b8f549-78m8r"] Nov 03 02:01:08.596064 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:01:08.596032 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" path="/var/lib/kubelet/pods/a22fd95a-ae21-4278-bc3d-3e9e5afb1298/volumes" Nov 03 02:01:32.151995 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:01:32.151951 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:02:01.509167 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.509079 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 02:02:01.509739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.509289 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" podUID="42a3cc49-6241-4ba8-aaf7-3811848c699f" containerName="maas-api" containerID="cri-o://f46cabe976b4e128bf1ab03a4aeeb0c7c036853bc9d6998298bd1ba4f93c39ff" gracePeriod=30 Nov 03 02:02:01.700824 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.700785 4992 generic.go:356] "Generic (PLEG): container finished" podID="42a3cc49-6241-4ba8-aaf7-3811848c699f" containerID="f46cabe976b4e128bf1ab03a4aeeb0c7c036853bc9d6998298bd1ba4f93c39ff" exitCode=0 Nov 03 02:02:01.701175 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.701024 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" event={"ID":"42a3cc49-6241-4ba8-aaf7-3811848c699f","Type":"ContainerDied","Data":"f46cabe976b4e128bf1ab03a4aeeb0c7c036853bc9d6998298bd1ba4f93c39ff"} Nov 03 02:02:01.818477 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.818445 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 02:02:01.892334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.891290 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\") pod \"42a3cc49-6241-4ba8-aaf7-3811848c699f\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " Nov 03 02:02:01.892334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.891384 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\") pod \"42a3cc49-6241-4ba8-aaf7-3811848c699f\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " Nov 03 02:02:01.892334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.891420 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") pod \"42a3cc49-6241-4ba8-aaf7-3811848c699f\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " Nov 03 02:02:01.892334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.891499 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\" (UniqueName: \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\") pod \"42a3cc49-6241-4ba8-aaf7-3811848c699f\" (UID: \"42a3cc49-6241-4ba8-aaf7-3811848c699f\") " Nov 03 02:02:01.892957 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.892923 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca" pod "42a3cc49-6241-4ba8-aaf7-3811848c699f" (UID: "42a3cc49-6241-4ba8-aaf7-3811848c699f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:01.897407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.897359 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls" pod "42a3cc49-6241-4ba8-aaf7-3811848c699f" (UID: "42a3cc49-6241-4ba8-aaf7-3811848c699f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:01.898835 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.898804 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29" pod "42a3cc49-6241-4ba8-aaf7-3811848c699f" (UID: "42a3cc49-6241-4ba8-aaf7-3811848c699f"). InnerVolumeSpecName "kube-api-access-x6m29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:01.899123 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.899098 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs" pod "42a3cc49-6241-4ba8-aaf7-3811848c699f" (UID: "42a3cc49-6241-4ba8-aaf7-3811848c699f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:01.993453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.993408 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:01.993453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.993447 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a3cc49-6241-4ba8-aaf7-3811848c699f-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:01.993453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.993460 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6m29\" (UniqueName: \"kubernetes.io/projected/42a3cc49-6241-4ba8-aaf7-3811848c699f-kube-api-access-x6m29\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:01.993744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:01.993471 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a3cc49-6241-4ba8-aaf7-3811848c699f-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:02.151456 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:02:02.151369 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:02:02.709133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:02.709090 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" event={"ID":"42a3cc49-6241-4ba8-aaf7-3811848c699f","Type":"ContainerDied","Data":"516a6158e902b6a431230dcedc075654effa563ae1e1811e4b1af796092968d3"} Nov 03 02:02:02.709616 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:02.709158 4992 scope.go:122] "RemoveContainer" containerID="f46cabe976b4e128bf1ab03a4aeeb0c7c036853bc9d6998298bd1ba4f93c39ff" Nov 03 02:02:02.709616 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:02.709106 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8" Nov 03 02:02:02.883102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:02.883004 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 02:02:02.934954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:02.934914 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a00717-7c65464895-92zt8"] Nov 03 02:02:03.691099 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:03.691060 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 02:02:03.691296 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:03.691273 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" podUID="9d5ad328-db1e-434e-8519-dcd8f091e48e" containerName="maas-api" containerID="cri-o://27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a" gracePeriod=30 Nov 03 02:02:03.991108 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:03.991077 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 02:02:04.119076 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.119037 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\" (UniqueName: \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\") pod \"9d5ad328-db1e-434e-8519-dcd8f091e48e\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " Nov 03 02:02:04.119276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.119086 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\") pod \"9d5ad328-db1e-434e-8519-dcd8f091e48e\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " Nov 03 02:02:04.119276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.119251 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\") pod \"9d5ad328-db1e-434e-8519-dcd8f091e48e\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " Nov 03 02:02:04.119405 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.119317 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\") pod \"9d5ad328-db1e-434e-8519-dcd8f091e48e\" (UID: \"9d5ad328-db1e-434e-8519-dcd8f091e48e\") " Nov 03 02:02:04.119940 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.119910 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca" pod "9d5ad328-db1e-434e-8519-dcd8f091e48e" (UID: "9d5ad328-db1e-434e-8519-dcd8f091e48e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:04.121978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.121944 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5" pod "9d5ad328-db1e-434e-8519-dcd8f091e48e" (UID: "9d5ad328-db1e-434e-8519-dcd8f091e48e"). InnerVolumeSpecName "kube-api-access-zwfd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:04.124827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.124783 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls" pod "9d5ad328-db1e-434e-8519-dcd8f091e48e" (UID: "9d5ad328-db1e-434e-8519-dcd8f091e48e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:04.124968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.124820 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs" pod "9d5ad328-db1e-434e-8519-dcd8f091e48e" (UID: "9d5ad328-db1e-434e-8519-dcd8f091e48e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:02:04.220730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.220637 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfd5\" (UniqueName: \"kubernetes.io/projected/9d5ad328-db1e-434e-8519-dcd8f091e48e-kube-api-access-zwfd5\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:04.220730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.220674 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:04.220730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.220683 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5ad328-db1e-434e-8519-dcd8f091e48e-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:04.220730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.220694 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5ad328-db1e-434e-8519-dcd8f091e48e-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:04.595041 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.594942 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a3cc49-6241-4ba8-aaf7-3811848c699f" path="/var/lib/kubelet/pods/42a3cc49-6241-4ba8-aaf7-3811848c699f/volumes" Nov 03 02:02:04.726295 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.726262 4992 generic.go:356] "Generic (PLEG): container finished" podID="9d5ad328-db1e-434e-8519-dcd8f091e48e" containerID="27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a" exitCode=0 Nov 03 02:02:04.726478 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.726313 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" Nov 03 02:02:04.726478 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.726346 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" event={"ID":"9d5ad328-db1e-434e-8519-dcd8f091e48e","Type":"ContainerDied","Data":"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a"} Nov 03 02:02:04.726478 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.726402 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg" event={"ID":"9d5ad328-db1e-434e-8519-dcd8f091e48e","Type":"ContainerDied","Data":"93fcf763cef14963b1e83e8ccddd034de5be54c7f013483516b1effba104df39"} Nov 03 02:02:04.726478 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.726427 4992 scope.go:122] "RemoveContainer" containerID="27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a" Nov 03 02:02:04.736714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.736689 4992 scope.go:122] "RemoveContainer" containerID="27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a" Nov 03 02:02:04.737080 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:02:04.737057 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a\": container with ID starting with 27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a not found: ID does not exist" containerID="27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a" Nov 03 02:02:04.737143 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.737092 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a"} err="failed to get container status \"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a\": rpc error: code = NotFound desc = could not find container \"27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a\": container with ID starting with 27b2577601d7639d3fbcb33a11f53e29df228814b1bd5d305e44b3abffe6819a not found: ID does not exist" Nov 03 02:02:04.897022 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.896924 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 02:02:04.901108 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:04.901072 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f95773-784dd5756f-ghvkg"] Nov 03 02:02:06.594454 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:06.594419 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5ad328-db1e-434e-8519-dcd8f091e48e" path="/var/lib/kubelet/pods/9d5ad328-db1e-434e-8519-dcd8f091e48e/volumes" Nov 03 02:02:11.953725 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:11.953681 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 02:02:11.954126 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:11.953972 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" containerID="cri-o://ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684" gracePeriod=30 Nov 03 02:02:17.225249 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.225172 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 02:02:17.272228 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272188 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272281 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272303 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272379 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272408 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272435 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272464 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272500 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272499 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272529 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\") pod \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\" (UID: \"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e\") " Nov 03 02:02:17.272710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272655 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:17.273034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272787 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:17.273034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272925 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:02:17.273034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.272933 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:17.273153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.273079 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.273153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.273100 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.273153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.273116 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.273153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.273131 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.273153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.273146 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.274792 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.274765 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:17.274978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.274958 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:02:17.275214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.275189 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:02:17.275214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.275195 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d" pod "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" (UID: "306a0bde-f8a0-4227-8a3f-ca9411d9ba1e"). InnerVolumeSpecName "kube-api-access-42q8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:02:17.374757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.374716 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.374757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.374749 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.374757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.374759 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.375012 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.374768 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42q8d\" (UniqueName: \"kubernetes.io/projected/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e-kube-api-access-42q8d\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:02:17.803918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.803880 4992 generic.go:356] "Generic (PLEG): container finished" podID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerID="ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684" exitCode=0 Nov 03 02:02:17.804115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.803935 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" Nov 03 02:02:17.804115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.803958 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerDied","Data":"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684"} Nov 03 02:02:17.804115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.804017 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct" event={"ID":"306a0bde-f8a0-4227-8a3f-ca9411d9ba1e","Type":"ContainerDied","Data":"9ccaa95a2c0e955a982f89dcbbb0e58bdb74e58fdf9f65a1d1a1c708d82f1b2b"} Nov 03 02:02:17.804115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.804040 4992 scope.go:122] "RemoveContainer" containerID="ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684" Nov 03 02:02:17.816685 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.816653 4992 scope.go:122] "RemoveContainer" containerID="0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6" Nov 03 02:02:17.825853 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.825833 4992 scope.go:122] "RemoveContainer" containerID="ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684" Nov 03 02:02:17.826157 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:02:17.826136 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684\": container with ID starting with ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684 not found: ID does not exist" containerID="ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684" Nov 03 02:02:17.826226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.826168 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684"} err="failed to get container status \"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684\": rpc error: code = NotFound desc = could not find container \"ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684\": container with ID starting with ccf599b582fd99665a5fcd7776cac61fd5a52803e810a3a7c68a5c63a335b684 not found: ID does not exist" Nov 03 02:02:17.826226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.826185 4992 scope.go:122] "RemoveContainer" containerID="0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6" Nov 03 02:02:17.826464 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:02:17.826443 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6\": container with ID starting with 0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6 not found: ID does not exist" containerID="0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6" Nov 03 02:02:17.826504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.826470 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6"} err="failed to get container status \"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6\": rpc error: code = NotFound desc = could not find container \"0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6\": container with ID starting with 0e296e25435e750f863e065c5b61d683cc4d979db7952ebce4fee037eaeeefb6 not found: ID does not exist" Nov 03 02:02:17.977918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:17.977873 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 02:02:18.029368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:18.029333 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a00717-openshift-default-8666f9c79c-94hct"] Nov 03 02:02:18.595612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:18.595576 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" path="/var/lib/kubelet/pods/306a0bde-f8a0-4227-8a3f-ca9411d9ba1e/volumes" Nov 03 02:02:32.151046 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:02:32.151007 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:02:44.953385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:44.953345 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:02:44.953974 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:02:44.953924 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:03:02.152417 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:03:02.152371 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:03:22.929353 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929320 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929824 4992 cpu_manager.go:414] "containerMap: removing container" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929837 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929854 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9d5ad328-db1e-434e-8519-dcd8f091e48e" containerName="maas-api" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929859 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5ad328-db1e-434e-8519-dcd8f091e48e" containerName="maas-api" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929873 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929880 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929886 4992 cpu_manager.go:414] "containerMap: removing container" podUID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" containerName="maas-api" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929891 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" containerName="maas-api" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929897 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929902 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929908 4992 cpu_manager.go:414] "containerMap: removing container" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.929909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929913 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929921 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929926 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929931 4992 cpu_manager.go:414] "containerMap: removing container" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929936 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929945 4992 cpu_manager.go:414] "containerMap: removing container" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929950 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929956 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929960 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929970 4992 cpu_manager.go:414] "containerMap: removing container" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929974 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929981 4992 cpu_manager.go:414] "containerMap: removing container" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929986 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929994 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.929999 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930008 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930012 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930022 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930026 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930032 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930036 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930049 4992 cpu_manager.go:414] "containerMap: removing container" podUID="42a3cc49-6241-4ba8-aaf7-3811848c699f" containerName="maas-api" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930055 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a3cc49-6241-4ba8-aaf7-3811848c699f" containerName="maas-api" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930061 4992 cpu_manager.go:414] "containerMap: removing container" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930065 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930137 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930144 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef0bc65-ba8a-4e7f-878f-530c05131b6d" containerName="maas-api" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930152 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930159 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930165 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930171 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930179 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930184 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930190 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930196 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ef3c61-064e-4c42-93ce-c55c11aabb79" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930202 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930208 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22fd95a-ae21-4278-bc3d-3e9e5afb1298" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930214 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3e5a8b-ce96-4eb9-a1df-4926b5eed06f" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930221 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a3cc49-6241-4ba8-aaf7-3811848c699f" containerName="maas-api" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930227 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="306a0bde-f8a0-4227-8a3f-ca9411d9ba1e" containerName="istio-proxy" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930232 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:22.930371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.930238 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5ad328-db1e-434e-8519-dcd8f091e48e" containerName="maas-api" Nov 03 02:03:22.933901 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:22.933884 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.026875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.026835 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:03:23.049422 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049381 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049437 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049542 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049626 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049715 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049776 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049873 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.049917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049900 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.050018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.049945 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zv2\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151119 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151164 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151209 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151243 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151315 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151348 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151397 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zv2\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151465 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151518 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151638 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151886 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151753 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.151886 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.151845 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.152028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.152005 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.152089 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.152032 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.153818 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.153792 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.153980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.153964 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.214397 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.214317 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.214397 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.214379 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zv2\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.248195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.248160 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:23.675694 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:03:23.675660 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1c1012_898b_469f_87e1_79cf30f11b25.slice/crio-bca7702f6ae1fe5ebd70edaf962b66e1ec930819297ce835cf6a27aaa92980fc WatchSource:0}: Error finding container bca7702f6ae1fe5ebd70edaf962b66e1ec930819297ce835cf6a27aaa92980fc: Status 404 returned error can't find the container with id bca7702f6ae1fe5ebd70edaf962b66e1ec930819297ce835cf6a27aaa92980fc Nov 03 02:03:23.678444 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.678418 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:03:23.678552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.678412 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:23.678552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.678521 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:23.678709 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:23.678590 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:24.207173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.207088 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" event={"ID":"6a1c1012-898b-469f-87e1-79cf30f11b25","Type":"ContainerStarted","Data":"1cde91bdbba3338478513ff2aefc100c01147272d9608289015318b826501de1"} Nov 03 02:03:24.207173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.207140 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" event={"ID":"6a1c1012-898b-469f-87e1-79cf30f11b25","Type":"ContainerStarted","Data":"bca7702f6ae1fe5ebd70edaf962b66e1ec930819297ce835cf6a27aaa92980fc"} Nov 03 02:03:24.248695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.248650 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:24.250859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.250830 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:24.251022 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.250884 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:24.333765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:24.333685 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podStartSLOduration=2.333672147 podStartE2EDuration="2.333672147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:24.331346604 +0000 UTC m=+2739.910169189" watchObservedRunningTime="2026-11-03 02:03:24.333672147 +0000 UTC m=+2739.912494732" Nov 03 02:03:25.249007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:25.248969 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:25.249425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:25.249025 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:26.249082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:26.249047 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:26.249487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:26.249102 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:27.249467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:27.249429 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:27.250014 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:27.249506 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:28.248572 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:28.248509 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:28.248766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:28.248602 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:29.248768 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:29.248732 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:29.249199 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:29.248791 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:30.249245 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:30.249210 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 02:03:30.249824 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:30.249281 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 02:03:32.151928 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:03:32.151875 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:03:32.289819 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:32.289766 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:03:32.290044 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:32.289853 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:03:32.389387 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:32.389353 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:32.389870 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:32.389855 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:32.390673 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:32.390648 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:03:49.212318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.212279 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:03:49.212859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.212840 4992 cpu_manager.go:414] "containerMap: removing container" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:49.212909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.212861 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:49.212979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.212969 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="129b9fd5-f0c1-483a-b391-c214c014ef52" containerName="cleanup" Nov 03 02:03:49.213019 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.212983 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63c8b47-42be-4bda-8d07-36df21a05d2d" containerName="cleanup" Nov 03 02:03:49.216358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.216336 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.319650 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321253 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321342 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321369 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2bl\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321438 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321489 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321585 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321611 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321653 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.323614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.321738 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422443 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422484 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2bl\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422529 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422602 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422675 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422713 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.422785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422749 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422834 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.422891 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.423346 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423586 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.423425 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.423467 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.423503 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.423855 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.423616 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.425393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.425371 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.425830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.425809 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.486543 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.486453 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.488733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.488692 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2bl\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.535793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.535752 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:49.963022 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.962984 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:03:49.963627 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.963553 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:49.963706 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.963688 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:49.963750 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:49.963727 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:03:50.375772 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:50.375681 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" event={"ID":"7029f41c-6dc0-4dc1-9bfc-808b89943f33","Type":"ContainerStarted","Data":"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a"} Nov 03 02:03:50.375772 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:50.375750 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" event={"ID":"7029f41c-6dc0-4dc1-9bfc-808b89943f33","Type":"ContainerStarted","Data":"1d2b55aa3d3d91095425f797e4e55d267667dc6b960b1c050f731ec6cec39e54"} Nov 03 02:03:50.501036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:50.500985 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podStartSLOduration=1.5009713489999998 podStartE2EDuration="1.500971349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:50.499907032 +0000 UTC m=+2766.078729612" watchObservedRunningTime="2026-11-03 02:03:50.500971349 +0000 UTC m=+2766.079793932" Nov 03 02:03:50.536229 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:50.536188 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:51.538234 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:51.538194 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:51.538663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:51.538255 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:52.536644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:52.536609 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:52.536832 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:52.536667 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:53.536784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:53.536742 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:53.537221 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:53.536826 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:54.536074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:54.536034 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:54.536362 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:54.536102 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:55.536583 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:55.536428 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:55.536583 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:55.536495 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:56.536292 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:56.536248 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 02:03:56.536465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:56.536331 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 02:03:57.796354 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:57.796321 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:57.797085 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:57.796960 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:03:57.797718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:03:57.797688 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:04:02.151681 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:04:02.151643 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:04:04.200037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.199998 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:04:04.204091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.204069 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.225767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.225730 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.225767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.225770 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.225977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.225795 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.225977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.225954 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.226046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.225990 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.226046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.226022 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.226172 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.226151 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.226208 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.226189 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.226313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.226301 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlrh\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.314523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.314480 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:04:04.327588 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327536 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlrh\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327600 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327625 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327649 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327708 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327729 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.327755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327753 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327831 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.327871 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.328186 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.328264 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.328311 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328399 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.328381 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.328644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.328626 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.330130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.330110 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.330736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.330716 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.390526 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.390485 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.390695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.390629 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlrh\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.519805 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.519712 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:04.944613 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:04:04.944541 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c61fe_a79d_449e_be65_56dae3e60e68.slice/crio-d76582aa434babe6165a8f84e3cfe99cf78965460f0e0b42564840174bbe47ba WatchSource:0}: Error finding container d76582aa434babe6165a8f84e3cfe99cf78965460f0e0b42564840174bbe47ba: Status 404 returned error can't find the container with id d76582aa434babe6165a8f84e3cfe99cf78965460f0e0b42564840174bbe47ba Nov 03 02:04:04.947730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.947690 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:04.947831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.947731 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:04:04.947831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.947768 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:04.947831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:04.947806 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:05.504648 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:05.504598 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" event={"ID":"4f5c61fe-a79d-449e-be65-56dae3e60e68","Type":"ContainerStarted","Data":"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a"} Nov 03 02:04:05.505066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:05.504666 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" event={"ID":"4f5c61fe-a79d-449e-be65-56dae3e60e68","Type":"ContainerStarted","Data":"d76582aa434babe6165a8f84e3cfe99cf78965460f0e0b42564840174bbe47ba"} Nov 03 02:04:05.520260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:05.520227 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:05.629387 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:05.629332 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podStartSLOduration=1.62931663 podStartE2EDuration="1.62931663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:05.62781472 +0000 UTC m=+2781.206637308" watchObservedRunningTime="2026-11-03 02:04:05.62931663 +0000 UTC m=+2781.208139216" Nov 03 02:04:06.522226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:06.522190 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:06.522656 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:06.522246 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:07.520755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:07.520714 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:07.520967 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:07.520781 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:08.520760 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:08.520722 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:08.521271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:08.520791 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:09.520129 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:09.520092 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:09.520320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:09.520148 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:10.520244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:10.520199 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:10.520712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:10.520268 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:11.520472 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:11.520429 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 02:04:11.520958 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:11.520501 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 02:04:12.585416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:12.585383 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:12.586255 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:12.586229 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:12.595259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:12.595230 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:04:23.391096 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.391059 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:04:23.394986 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.394967 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.442895 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.442854 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:04:23.446791 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.446758 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.446963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.446822 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.446963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.446829 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.446963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.446847 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkkj\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.446963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.446866 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.447165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.447031 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.447165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.447062 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.447165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.447143 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.447165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.447165 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.447333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.447198 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.527544 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.527508 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:04:23.548416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548359 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548460 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548511 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548611 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztwlw\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548648 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkkj\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548665 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.548684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548681 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548699 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548741 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548773 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548814 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548818 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548924 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548960 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.548997 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549010 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549031 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549098 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549320 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549348 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.549584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549464 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549503 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.549877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.549588 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.551418 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.551394 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.551706 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.551681 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.597649 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.597607 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:04:23.611761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.611731 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.612010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.611985 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkkj\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.650361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650267 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650545 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650475 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztwlw\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650545 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650526 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650709 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650586 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650709 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650695 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650768 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650908 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650955 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.650999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.650990 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.651112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.651068 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.651112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.651085 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.651280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.651258 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.651371 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.651352 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.651426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.651377 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.653063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.653043 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.653357 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.653335 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.709235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.709197 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:23.713373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.713336 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.714001 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.713623 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztwlw\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:23.761421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:23.761384 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:24.135952 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.135912 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:04:24.169283 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:04:24.169241 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c2e0d7_c418_4e94_98f4_874b193e917d.slice/crio-d6751037d7b862be61b8b81ad84ef3ea080db68ed7ce0b29d2efea6a188d7424 WatchSource:0}: Error finding container d6751037d7b862be61b8b81ad84ef3ea080db68ed7ce0b29d2efea6a188d7424: Status 404 returned error can't find the container with id d6751037d7b862be61b8b81ad84ef3ea080db68ed7ce0b29d2efea6a188d7424 Nov 03 02:04:24.171875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.171840 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.171966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.171909 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.171966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.171948 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.185283 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:04:24.185252 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132fe9d0_a453_440b_946c_862a26f3c87f.slice/crio-2186cca9e18f773aa214b3e7e76ea7beee2486e0280de3a55f8bd8fb8a92f9c2 WatchSource:0}: Error finding container 2186cca9e18f773aa214b3e7e76ea7beee2486e0280de3a55f8bd8fb8a92f9c2: Status 404 returned error can't find the container with id 2186cca9e18f773aa214b3e7e76ea7beee2486e0280de3a55f8bd8fb8a92f9c2 Nov 03 02:04:24.187570 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.187531 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:04:24.187952 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.187914 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.188020 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.188002 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.188060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.188042 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:04:24.690611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.690570 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" event={"ID":"132fe9d0-a453-440b-946c-862a26f3c87f","Type":"ContainerStarted","Data":"3bad48d76aeeeb24511f90a97246c9d5ecc260931807e23c75da6e31b89eb3e0"} Nov 03 02:04:24.691075 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.690640 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" event={"ID":"132fe9d0-a453-440b-946c-862a26f3c87f","Type":"ContainerStarted","Data":"2186cca9e18f773aa214b3e7e76ea7beee2486e0280de3a55f8bd8fb8a92f9c2"} Nov 03 02:04:24.692329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.692301 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" event={"ID":"79c2e0d7-c418-4e94-98f4-874b193e917d","Type":"ContainerStarted","Data":"7ca9f1e66a50d33a1942700c22179862908133c8661d7316442c1c361b33db7a"} Nov 03 02:04:24.692438 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.692346 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" event={"ID":"79c2e0d7-c418-4e94-98f4-874b193e917d","Type":"ContainerStarted","Data":"d6751037d7b862be61b8b81ad84ef3ea080db68ed7ce0b29d2efea6a188d7424"} Nov 03 02:04:24.710143 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.710092 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:24.762127 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.762090 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:24.818740 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.818672 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podStartSLOduration=1.8186495539999998 podStartE2EDuration="1.818649554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:24.814609351 +0000 UTC m=+2800.393431939" watchObservedRunningTime="2026-11-03 02:04:24.818649554 +0000 UTC m=+2800.397472144" Nov 03 02:04:24.944192 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:24.944081 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podStartSLOduration=1.944067733 podStartE2EDuration="1.944067733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:24.941166546 +0000 UTC m=+2800.519989132" watchObservedRunningTime="2026-11-03 02:04:24.944067733 +0000 UTC m=+2800.522890320" Nov 03 02:04:25.718420 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:25.718385 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:25.718926 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:25.718895 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:25.799692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:25.787720 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:25.799692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:25.787788 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:26.710531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:26.710039 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:26.710531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:26.710120 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:26.782585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:26.782318 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:26.782585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:26.782390 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:27.711115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:27.711076 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:27.711406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:27.711363 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:27.773585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:27.765980 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:27.773585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:27.766051 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:28.711899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:28.711860 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:28.722821 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:28.722779 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:28.763094 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:28.763030 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:28.771580 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:28.763460 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:29.710585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:29.709948 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:29.710585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:29.710010 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:29.771594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:29.770980 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:29.771594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:29.771060 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:30.713499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:30.713455 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 02:04:30.713817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:30.713521 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 02:04:30.762596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:30.762501 4992 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 02:04:30.762818 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:30.762618 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 02:04:31.905659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.905609 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:31.906859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.906835 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:31.910201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.910168 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:31.910988 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.910958 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:04:31.911230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.911211 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:31.913393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:31.913372 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:04:32.155916 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:04:32.155827 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:04:33.667685 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.667650 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:04:33.672774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.672749 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.727258 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.727188 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f7f61d-serving-cert\"" Nov 03 02:04:33.727645 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.727608 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:04:33.762883 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.762848 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.763128 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.763019 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8sz\" (UniqueName: \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.763203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.763154 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.763256 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.763232 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.865245 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.864920 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8sz\" (UniqueName: \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.865411 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.865337 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.865517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.865434 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.865596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.865542 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.865689 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:04:33.865660 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-serving-cert: secret "maas-api-e2e-praxis-b-f7f61d-serving-cert" not found Nov 03 02:04:33.865761 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:04:33.865748 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls podName:17728b2d-56df-40b2-8842-4085dbd92a29 nodeName:}" failed. No retries permitted until 2026-11-03 02:04:34.365724786 +0000 UTC m=+2809.944547359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls") pod "maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" (UID: "17728b2d-56df-40b2-8842-4085dbd92a29") : secret "maas-api-e2e-praxis-b-f7f61d-serving-cert" not found Nov 03 02:04:33.866092 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.866067 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.868098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.868074 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:33.932090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:33.931960 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8sz\" (UniqueName: \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:34.371324 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:34.371235 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:34.373797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:34.373763 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:34.588184 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:34.588141 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:34.985711 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:04:34.985664 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17728b2d_56df_40b2_8842_4085dbd92a29.slice/crio-05001e78c4781f9b399dee31c1fc3deb7c7a17588fb1144144d9cee5c2f37d33 WatchSource:0}: Error finding container 05001e78c4781f9b399dee31c1fc3deb7c7a17588fb1144144d9cee5c2f37d33: Status 404 returned error can't find the container with id 05001e78c4781f9b399dee31c1fc3deb7c7a17588fb1144144d9cee5c2f37d33 Nov 03 02:04:34.986124 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:34.985941 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:04:34.987073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:34.987052 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:04:35.783079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:35.783029 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" event={"ID":"17728b2d-56df-40b2-8842-4085dbd92a29","Type":"ContainerStarted","Data":"05001e78c4781f9b399dee31c1fc3deb7c7a17588fb1144144d9cee5c2f37d33"} Nov 03 02:04:36.794158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:36.794034 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" event={"ID":"17728b2d-56df-40b2-8842-4085dbd92a29","Type":"ContainerStarted","Data":"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de"} Nov 03 02:04:36.794671 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:36.794488 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:04:36.917528 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:36.917469 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" podStartSLOduration=2.860629295 podStartE2EDuration="3.917452963s" totalImagesPullingTime="1.056823668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:36.91551292 +0000 UTC m=+2812.494335498" watchObservedRunningTime="2026-11-03 02:04:36.917452963 +0000 UTC m=+2812.496275548" Nov 03 02:04:42.804794 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:04:42.804763 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:05:02.152165 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:05:02.152072 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:05:02.969184 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:02.969145 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:05:02.975334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:02.975306 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.029299 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.029263 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:05:03.077054 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.077020 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c34879-serving-cert\"" Nov 03 02:05:03.155764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.155727 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.156191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.155784 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.156191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.155881 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.156191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.155928 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p6q\" (UniqueName: \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.257441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.257353 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.257441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.257415 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.257683 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.257456 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.257683 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.257496 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p6q\" (UniqueName: \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.258102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.258080 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.260100 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.260080 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.260162 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.260146 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.321229 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.321192 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p6q\" (UniqueName: \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\") pod \"maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.588972 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.588889 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:03.963488 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:05:03.963459 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1594ec_aadb_4907_8b86_a1c1894d2969.slice/crio-14034736769eb56e5794bb8fc76f0973536fc179a638f5bfca2cac4ff9a3c216 WatchSource:0}: Error finding container 14034736769eb56e5794bb8fc76f0973536fc179a638f5bfca2cac4ff9a3c216: Status 404 returned error can't find the container with id 14034736769eb56e5794bb8fc76f0973536fc179a638f5bfca2cac4ff9a3c216 Nov 03 02:05:03.966327 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:03.966300 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:05:04.966473 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:04.966436 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" event={"ID":"dd1594ec-aadb-4907-8b86-a1c1894d2969","Type":"ContainerStarted","Data":"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c"} Nov 03 02:05:04.967021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:04.966500 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" event={"ID":"dd1594ec-aadb-4907-8b86-a1c1894d2969","Type":"ContainerStarted","Data":"14034736769eb56e5794bb8fc76f0973536fc179a638f5bfca2cac4ff9a3c216"} Nov 03 02:05:04.967021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:04.966699 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:05.095152 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:05.095011 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" podStartSLOduration=2.232386243 podStartE2EDuration="3.094779376s" totalImagesPullingTime="862.393133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:05.092244546 +0000 UTC m=+2840.671067133" watchObservedRunningTime="2026-11-03 02:05:05.094779376 +0000 UTC m=+2840.673601964" Nov 03 02:05:10.977991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:10.977953 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:05:32.152953 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:05:32.152910 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:05:50.771331 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.771288 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:05:50.776415 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.776386 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.831740 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.831694 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-1c9b4e-openshift-default-dockercfg-t2wch\"" Nov 03 02:05:50.880015 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.879975 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880221 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880084 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880221 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880202 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880240 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880297 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880315 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880416 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880470 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pzb\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.880672 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.880530 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.887529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.887491 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:05:50.982232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982194 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982292 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982382 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982534 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982754 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982731 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982839 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982783 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982839 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982808 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982862 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.982954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.982894 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pzb\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.983054 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.983023 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.983351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.983325 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.983505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.983445 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.983925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.983893 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.984079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.983946 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.985738 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.985686 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:50.985996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:50.985973 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:51.045550 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.045470 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:51.046006 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.045982 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pzb\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\") pod \"e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:51.090992 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.090946 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:51.547520 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.547458 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:05:51.553753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.553710 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:51.553903 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.553803 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:51.553903 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:51.553849 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:05:52.297352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:52.297308 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerStarted","Data":"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed"} Nov 03 02:05:52.297789 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:52.297369 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerStarted","Data":"3df3d075d9f1b7e676f3600ba82858818d2774f225072d8b206a7f12aaca423b"} Nov 03 02:05:52.422535 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:52.422484 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podStartSLOduration=2.422470907 podStartE2EDuration="2.422470907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:52.421377289 +0000 UTC m=+2888.000199876" watchObservedRunningTime="2026-11-03 02:05:52.422470907 +0000 UTC m=+2888.001293494" Nov 03 02:05:53.091964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:53.091923 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:53.098247 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:53.098217 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:53.303530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:53.303494 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:05:53.304687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:05:53.304669 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:06:02.152393 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:06:02.152353 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:06:10.920967 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:10.920931 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:06:10.924961 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:10.924940 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.026511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.004232 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d9576a-openshift-default-dockercfg-k8ph4\"" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070406 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070455 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrq4s\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070517 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070534 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070573 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070613 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070635 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070685 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.071800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.070729 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.091627 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.084841 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:06:11.172364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172271 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172320 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172354 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172398 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172438 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172506 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172621 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172740 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172788 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172790 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq4s\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.172916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.172817 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.173222 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.173173 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.173283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.173227 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.173512 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.173473 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.175865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.175837 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.176020 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.175998 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.257927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.257873 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq4s\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.258302 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.258277 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\") pod \"e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.553884 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.553790 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:11.987674 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:06:11.987631 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacff4567_6d9a_40f4_9349_ac43f440dba4.slice/crio-c07e16e864ac4f211cb727a21246e932ce5167f9938ab813da12adbcb054fb5d WatchSource:0}: Error finding container c07e16e864ac4f211cb727a21246e932ce5167f9938ab813da12adbcb054fb5d: Status 404 returned error can't find the container with id c07e16e864ac4f211cb727a21246e932ce5167f9938ab813da12adbcb054fb5d Nov 03 02:06:11.990116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.990091 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:06:11.990971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.990932 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:11.991033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.991010 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:11.991069 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:11.991040 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:12.421805 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:12.421769 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerStarted","Data":"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d"} Nov 03 02:06:12.421999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:12.421825 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerStarted","Data":"c07e16e864ac4f211cb727a21246e932ce5167f9938ab813da12adbcb054fb5d"} Nov 03 02:06:12.549495 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:12.549438 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podStartSLOduration=2.54942244 podStartE2EDuration="2.54942244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:12.546704613 +0000 UTC m=+2908.125527200" watchObservedRunningTime="2026-11-03 02:06:12.54942244 +0000 UTC m=+2908.128245028" Nov 03 02:06:12.553883 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:12.553847 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:13.565451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:13.565392 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:14.433301 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:14.433264 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:14.434442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:14.434422 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:06:25.874521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:25.874428 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:06:25.879162 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:25.879134 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.016304 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.007017 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6a5d73-openshift-default-dockercfg-jh8dw\"" Nov 03 02:06:26.045600 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.039052 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:06:26.140843 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.140742 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.140942 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141012 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141045 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141207 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4jc\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141259 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141287 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141323 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.141445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.141350 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.242744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.242704 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.242973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.242751 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.242973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.242785 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.242973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.242817 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.242973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.242911 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243213 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243037 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243213 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243104 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243213 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243130 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243362 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243220 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243362 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243236 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4jc\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243664 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243692 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243901 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243756 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.243901 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.243877 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.245517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.245491 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.245701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.245673 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.315279 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.314287 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.315279 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.314389 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4jc\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\") pod \"e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.493942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.493888 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:26.924122 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:06:26.924075 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8acc2db_ccfa_4d0f_b206_c5d40224deae.slice/crio-2d8f1db6ef1be1e8b840b4f439fdfaf7c71594951ebd94dd2a2fae0da14b54af WatchSource:0}: Error finding container 2d8f1db6ef1be1e8b840b4f439fdfaf7c71594951ebd94dd2a2fae0da14b54af: Status 404 returned error can't find the container with id 2d8f1db6ef1be1e8b840b4f439fdfaf7c71594951ebd94dd2a2fae0da14b54af Nov 03 02:06:26.927076 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.927050 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:06:26.927309 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.927269 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:26.927379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.927361 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:26.927423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:26.927407 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:06:27.521601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:27.521541 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerStarted","Data":"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a"} Nov 03 02:06:27.521804 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:27.521616 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerStarted","Data":"2d8f1db6ef1be1e8b840b4f439fdfaf7c71594951ebd94dd2a2fae0da14b54af"} Nov 03 02:06:27.648406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:27.648351 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podStartSLOduration=2.648335975 podStartE2EDuration="2.648335975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:27.6448611 +0000 UTC m=+2923.223683686" watchObservedRunningTime="2026-11-03 02:06:27.648335975 +0000 UTC m=+2923.227158561" Nov 03 02:06:28.494813 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:28.494775 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:28.501825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:28.501799 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:28.532333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:28.532299 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:28.533376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:06:28.533351 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:06:32.151375 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:06:32.151329 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:07:02.152497 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:07:02.152446 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:07:32.152961 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:07:32.152909 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:07:33.964334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.964295 4992 generic.go:356] "Generic (PLEG): container finished" podID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerID="f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed" exitCode=0 Nov 03 02:07:33.964855 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.964377 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerDied","Data":"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed"} Nov 03 02:07:33.964901 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.964877 4992 scope.go:122] "RemoveContainer" containerID="f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed" Nov 03 02:07:33.966396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.966366 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:33.966455 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.966441 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:33.966489 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:33.966474 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:34.992511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:34.992479 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerStarted","Data":"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27"} Nov 03 02:07:35.091186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:35.091154 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:07:35.092716 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:35.092688 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:35.092842 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:35.092737 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:36.092168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:36.092132 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:36.092611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:36.092186 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:37.091608 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:37.091532 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:37.091802 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:37.091646 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:38.092364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:38.092324 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:38.092836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:38.092384 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:38.303767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:38.303728 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:07:39.091865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:39.091830 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:39.092057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:39.091885 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:40.091451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:40.091415 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 02:07:40.091916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:40.091472 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 02:07:40.990630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:40.990592 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:07:40.995116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:40.995097 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.049989 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.049951 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:07:41.072005 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.071969 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.072215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.072019 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.072215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.072113 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.072215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.072163 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqn6\" (UniqueName: \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.096337 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.096305 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:07:41.097283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.097262 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:07:41.098543 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.098529 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6a5d73-serving-cert\"" Nov 03 02:07:41.173305 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.173263 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.173487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.173359 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.173487 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:07:41.173418 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-serving-cert: secret "maas-api-e2e-shared-b-w2-6a5d73-serving-cert" not found Nov 03 02:07:41.173487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.173449 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqn6\" (UniqueName: \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.173682 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:07:41.173492 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls podName:e9fab307-193c-420b-bdcd-7a5e2c6d104e nodeName:}" failed. No retries permitted until 2026-11-03 02:07:41.673475599 +0000 UTC m=+2997.252298176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" (UID: "e9fab307-193c-420b-bdcd-7a5e2c6d104e") : secret "maas-api-e2e-shared-b-w2-6a5d73-serving-cert" not found Nov 03 02:07:41.174169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.174141 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.174994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.174969 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.176096 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.176067 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.238177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.238139 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqn6\" (UniqueName: \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.680983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.680948 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.683440 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.683410 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:41.909647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:41.909613 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:42.291785 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:07:42.291748 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fab307_193c_420b_bdcd_7a5e2c6d104e.slice/crio-63691a628aa61771460300a37e355d706ddc77c545d92ee87359e9c7af38bb83 WatchSource:0}: Error finding container 63691a628aa61771460300a37e355d706ddc77c545d92ee87359e9c7af38bb83: Status 404 returned error can't find the container with id 63691a628aa61771460300a37e355d706ddc77c545d92ee87359e9c7af38bb83 Nov 03 02:07:42.295131 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:42.295103 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:07:43.047327 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:43.047287 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" event={"ID":"e9fab307-193c-420b-bdcd-7a5e2c6d104e","Type":"ContainerStarted","Data":"63691a628aa61771460300a37e355d706ddc77c545d92ee87359e9c7af38bb83"} Nov 03 02:07:44.055334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:44.055295 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" event={"ID":"e9fab307-193c-420b-bdcd-7a5e2c6d104e","Type":"ContainerStarted","Data":"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61"} Nov 03 02:07:44.055759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:44.055363 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:44.176402 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:44.176347 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" podStartSLOduration=3.214304281 podStartE2EDuration="4.176334257s" totalImagesPullingTime="962.029976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:44.175843398 +0000 UTC m=+2999.754665985" watchObservedRunningTime="2026-11-03 02:07:44.176334257 +0000 UTC m=+2999.755156844" Nov 03 02:07:45.021527 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:45.021498 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:07:45.021527 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:45.021518 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:07:50.090696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:50.090661 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:07:55.155015 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.154980 4992 generic.go:356] "Generic (PLEG): container finished" podID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerID="4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d" exitCode=0 Nov 03 02:07:55.155472 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.155042 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerDied","Data":"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d"} Nov 03 02:07:55.155472 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.155468 4992 scope.go:122] "RemoveContainer" containerID="4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d" Nov 03 02:07:55.157063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.157028 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:55.157182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.157116 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:55.157182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:55.157165 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:07:56.176591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:56.169666 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerStarted","Data":"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497"} Nov 03 02:07:56.554164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:56.554072 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:07:56.555274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:56.555243 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:07:56.555403 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:56.555296 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 02:07:57.554460 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:57.554411 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:07:57.554898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:57.554496 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 02:07:58.554379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:58.554334 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:07:58.554581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:58.554398 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 02:07:59.434111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:59.434079 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:07:59.555130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:59.555092 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:07:59.555521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:07:59.555165 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 02:08:00.555231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:00.555174 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 02:08:00.555680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:00.555270 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 02:08:01.559728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:01.559693 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:08:01.560829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:01.560805 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:08:02.150897 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:08:02.150858 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:08:24.382750 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.382717 4992 generic.go:356] "Generic (PLEG): container finished" podID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerID="330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a" exitCode=0 Nov 03 02:08:24.383192 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.382790 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerDied","Data":"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a"} Nov 03 02:08:24.383246 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.383231 4992 scope.go:122] "RemoveContainer" containerID="330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a" Nov 03 02:08:24.384882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.384850 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:24.384950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.384918 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:24.384987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:24.384954 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:08:25.409520 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:25.409483 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerStarted","Data":"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b"} Nov 03 02:08:25.495098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:25.495053 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:08:25.496542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:25.496515 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:25.496690 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:25.496621 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:26.494468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:26.494432 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:26.494986 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:26.494495 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:27.495279 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:27.495238 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:27.495751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:27.495315 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:28.494583 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:28.494517 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:28.494770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:28.494614 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:28.532631 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:28.532592 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:08:29.494313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:29.494274 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:29.494516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:29.494334 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:30.495321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:30.495283 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 02:08:30.495774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:30.495352 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 02:08:31.499026 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:31.498993 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:08:31.500149 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:08:31.500129 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:08:32.151546 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:08:32.151504 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:09:02.152573 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:09:02.152521 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:09:31.239084 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:31.239043 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 02:09:31.239644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:31.239320 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" containerID="cri-o://dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af" gracePeriod=30 Nov 03 02:09:32.152876 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:09:32.152828 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:09:36.416782 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.416752 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:09:36.425982 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.424209 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.479062 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.479026 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:09:36.524097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524021 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524240 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524104 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsk2d\" (UniqueName: \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524240 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524141 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524261 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524281 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524312 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.524769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.524750 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 02:09:36.526532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.526498 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"kube-root-ca.crt\"" Nov 03 02:09:36.526635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.526513 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"openshift-service-ca.crt\"" Nov 03 02:09:36.526635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.526618 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"default-dockercfg-48fzx\"" Nov 03 02:09:36.532542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.532516 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-c34879\"/\"auth-test-model-c34879-kserve-self-signed-certs\"" Nov 03 02:09:36.625203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625165 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625224 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625306 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625329 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625374 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625414 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625496 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625611 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:36.625702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625675 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:36.625702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625693 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:36.625932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625677 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625801 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\") pod \"9db16e50-75ce-46cb-ac39-01e9b20e4625\" (UID: \"9db16e50-75ce-46cb-ac39-01e9b20e4625\") " Nov 03 02:09:36.625932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625860 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:09:36.625932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.625899 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:36.626144 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626114 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626162 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsk2d\" (UniqueName: \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626210 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626367 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626403 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626455 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626714 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626737 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626756 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626770 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626788 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db16e50-75ce-46cb-ac39-01e9b20e4625-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.626856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626808 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.627696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.626926 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.627696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.627374 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.628668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.628631 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "kube-api-access-vfc7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:09:36.629177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.629131 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.629349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.629327 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:09:36.629542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.629509 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.629656 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.629637 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:09:36.629867 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.629843 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token" pod "9db16e50-75ce-46cb-ac39-01e9b20e4625" (UID: "9db16e50-75ce-46cb-ac39-01e9b20e4625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:09:36.690066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.690027 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsk2d\" (UniqueName: \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\") pod \"auth-test-model-c34879-kserve-749cf9d44f-hqqhf\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.727895 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.727861 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.727895 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.727895 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfc7j\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-kube-api-access-vfc7j\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.728097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.727909 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.728097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.727925 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db16e50-75ce-46cb-ac39-01e9b20e4625-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:09:36.746722 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.746683 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:36.964852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.963404 4992 generic.go:356] "Generic (PLEG): container finished" podID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerID="dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af" exitCode=0 Nov 03 02:09:36.964852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.963613 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" event={"ID":"9db16e50-75ce-46cb-ac39-01e9b20e4625","Type":"ContainerDied","Data":"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af"} Nov 03 02:09:36.964852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.963662 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" event={"ID":"9db16e50-75ce-46cb-ac39-01e9b20e4625","Type":"ContainerDied","Data":"c1852fd4e88112562763d5a5e27fc1b31298e747cdb2feb98ced8f99e7a80113"} Nov 03 02:09:36.964852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.963691 4992 scope.go:122] "RemoveContainer" containerID="dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af" Nov 03 02:09:36.964852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.963861 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn" Nov 03 02:09:36.987661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.987638 4992 scope.go:122] "RemoveContainer" containerID="dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af" Nov 03 02:09:36.987976 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:09:36.987955 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af\": container with ID starting with dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af not found: ID does not exist" containerID="dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af" Nov 03 02:09:36.988023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:36.987991 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af"} err="failed to get container status \"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af\": rpc error: code = NotFound desc = could not find container \"dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af\": container with ID starting with dfea24bc97aafc96c270a5ba6cd35de7bc6492ea2d9a36f560c6cbe048a714af not found: ID does not exist" Nov 03 02:09:37.137910 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:37.137821 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 02:09:37.144766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:37.144720 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-74bnn"] Nov 03 02:09:37.147857 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:37.147831 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:09:37.183452 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:09:37.183418 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50f86c8c_d38b_4741_83f2_f1cd5821f70a.slice/crio-ae5c98ca69e8515678c2200726154b967055145c68c007b8aefbba55f5c5aa7f WatchSource:0}: Error finding container ae5c98ca69e8515678c2200726154b967055145c68c007b8aefbba55f5c5aa7f: Status 404 returned error can't find the container with id ae5c98ca69e8515678c2200726154b967055145c68c007b8aefbba55f5c5aa7f Nov 03 02:09:37.971999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:37.971953 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" event={"ID":"50f86c8c-d38b-4741-83f2-f1cd5821f70a","Type":"ContainerStarted","Data":"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850"} Nov 03 02:09:37.972417 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:37.972009 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" event={"ID":"50f86c8c-d38b-4741-83f2-f1cd5821f70a","Type":"ContainerStarted","Data":"ae5c98ca69e8515678c2200726154b967055145c68c007b8aefbba55f5c5aa7f"} Nov 03 02:09:38.093253 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:38.093197 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" podStartSLOduration=2.093179492 podStartE2EDuration="2.093179492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:38.092124166 +0000 UTC m=+3113.670946753" watchObservedRunningTime="2026-11-03 02:09:38.093179492 +0000 UTC m=+3113.672002079" Nov 03 02:09:38.595902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:38.595865 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" path="/var/lib/kubelet/pods/9db16e50-75ce-46cb-ac39-01e9b20e4625/volumes" Nov 03 02:09:38.747069 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:38.747031 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:46.747042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:46.747006 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:46.760149 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:46.760118 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:09:47.036442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:09:47.036355 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:10:02.152988 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:10:02.152943 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:10:04.681979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:04.681943 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:10:04.682546 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:04.682139 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" containerID="cri-o://21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a" gracePeriod=30 Nov 03 02:10:04.733638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:04.733599 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:10:04.733857 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:04.733835 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" containerID="cri-o://b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a" gracePeriod=30 Nov 03 02:10:10.017817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.017785 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:10:10.067872 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.067844 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:10:10.073687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073660 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.073786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073697 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.073786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073760 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.073906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073787 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.073906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073811 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.073906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073859 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.073906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073894 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073925 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073953 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.074115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.073964 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074022 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074325 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074124 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.074325 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.074325 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074115 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.074325 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074154 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.074325 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074261 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074321 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074335 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074417 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074461 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074547 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074611 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\") pod \"4f5c61fe-a79d-449e-be65-56dae3e60e68\" (UID: \"4f5c61fe-a79d-449e-be65-56dae3e60e68\") " Nov 03 02:10:10.074650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074622 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.075059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074650 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:10.075059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074652 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.075059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074676 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\") pod \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\" (UID: \"7029f41c-6dc0-4dc1-9bfc-808b89943f33\") " Nov 03 02:10:10.075059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.074921 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.075285 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075205 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:10.075344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075296 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075318 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075332 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075348 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075363 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075378 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075394 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c61fe-a79d-449e-be65-56dae3e60e68-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075407 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.075506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.075420 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.076535 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.076506 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "kube-api-access-2hlrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.076872 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.076851 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.077281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.077247 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:10.077388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.077305 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.077388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.077343 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "kube-api-access-2f2bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.077499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.077458 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:10.077678 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.077653 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo" pod "4f5c61fe-a79d-449e-be65-56dae3e60e68" (UID: "4f5c61fe-a79d-449e-be65-56dae3e60e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:10.078584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.078537 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy" pod "7029f41c-6dc0-4dc1-9bfc-808b89943f33" (UID: "7029f41c-6dc0-4dc1-9bfc-808b89943f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:10.176280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176238 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176273 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hlrh\" (UniqueName: \"kubernetes.io/projected/4f5c61fe-a79d-449e-be65-56dae3e60e68-kube-api-access-2hlrh\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176284 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176293 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176305 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176314 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176323 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7029f41c-6dc0-4dc1-9bfc-808b89943f33-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176332 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5c61fe-a79d-449e-be65-56dae3e60e68-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.176521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.176340 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2bl\" (UniqueName: \"kubernetes.io/projected/7029f41c-6dc0-4dc1-9bfc-808b89943f33-kube-api-access-2f2bl\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:10.183501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.183462 4992 generic.go:356] "Generic (PLEG): container finished" podID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerID="b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a" exitCode=0 Nov 03 02:10:10.183701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.183517 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" Nov 03 02:10:10.183701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.183551 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" event={"ID":"7029f41c-6dc0-4dc1-9bfc-808b89943f33","Type":"ContainerDied","Data":"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a"} Nov 03 02:10:10.183701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.183672 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft" event={"ID":"7029f41c-6dc0-4dc1-9bfc-808b89943f33","Type":"ContainerDied","Data":"1d2b55aa3d3d91095425f797e4e55d267667dc6b960b1c050f731ec6cec39e54"} Nov 03 02:10:10.183701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.183695 4992 scope.go:122] "RemoveContainer" containerID="b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a" Nov 03 02:10:10.185225 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.185198 4992 generic.go:356] "Generic (PLEG): container finished" podID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerID="21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a" exitCode=0 Nov 03 02:10:10.185326 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.185253 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" Nov 03 02:10:10.185326 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.185282 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" event={"ID":"4f5c61fe-a79d-449e-be65-56dae3e60e68","Type":"ContainerDied","Data":"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a"} Nov 03 02:10:10.185410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.185328 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd" event={"ID":"4f5c61fe-a79d-449e-be65-56dae3e60e68","Type":"ContainerDied","Data":"d76582aa434babe6165a8f84e3cfe99cf78965460f0e0b42564840174bbe47ba"} Nov 03 02:10:10.196675 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.196643 4992 scope.go:122] "RemoveContainer" containerID="b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a" Nov 03 02:10:10.197071 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:10:10.197042 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a\": container with ID starting with b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a not found: ID does not exist" containerID="b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a" Nov 03 02:10:10.197187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.197083 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a"} err="failed to get container status \"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a\": rpc error: code = NotFound desc = could not find container \"b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a\": container with ID starting with b8d569b5722ead2e6f6b8cb2945b254d1b77e086a60a6fda658c73ebb9a4bb3a not found: ID does not exist" Nov 03 02:10:10.197187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.197109 4992 scope.go:122] "RemoveContainer" containerID="21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a" Nov 03 02:10:10.208506 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.208482 4992 scope.go:122] "RemoveContainer" containerID="21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a" Nov 03 02:10:10.208884 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:10:10.208851 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a\": container with ID starting with 21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a not found: ID does not exist" containerID="21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a" Nov 03 02:10:10.208991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.208892 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a"} err="failed to get container status \"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a\": rpc error: code = NotFound desc = could not find container \"21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a\": container with ID starting with 21171f38d9b751c7483a34d57514ad16eea66d690787638201f5c857b53fd56a not found: ID does not exist" Nov 03 02:10:10.491542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.491508 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:10:10.496360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.496333 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-25bft"] Nov 03 02:10:10.595726 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.595683 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" path="/var/lib/kubelet/pods/7029f41c-6dc0-4dc1-9bfc-808b89943f33/volumes" Nov 03 02:10:10.610608 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.610549 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:10:10.614898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.614857 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-s7mcd"] Nov 03 02:10:10.991179 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.991140 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:10:10.991384 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.991365 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" containerID="cri-o://3bad48d76aeeeb24511f90a97246c9d5ecc260931807e23c75da6e31b89eb3e0" gracePeriod=30 Nov 03 02:10:10.996050 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.996018 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:10:10.996252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.996228 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" containerID="cri-o://7ca9f1e66a50d33a1942700c22179862908133c8661d7316442c1c361b33db7a" gracePeriod=30 Nov 03 02:10:10.998776 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.998748 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 02:10:10.998994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:10.998960 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" containerID="cri-o://a1e0e349fa4b2252045e4a1e877e2325d6e59c82302729873f86892f70b38b97" gracePeriod=30 Nov 03 02:10:11.042869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:11.042832 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:10:11.043379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:11.043044 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" containerID="cri-o://1cde91bdbba3338478513ff2aefc100c01147272d9608289015318b826501de1" gracePeriod=30 Nov 03 02:10:12.595577 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:12.595530 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" path="/var/lib/kubelet/pods/4f5c61fe-a79d-449e-be65-56dae3e60e68/volumes" Nov 03 02:10:16.238879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.238844 4992 generic.go:356] "Generic (PLEG): container finished" podID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerID="1cde91bdbba3338478513ff2aefc100c01147272d9608289015318b826501de1" exitCode=0 Nov 03 02:10:16.239358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.238951 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" event={"ID":"6a1c1012-898b-469f-87e1-79cf30f11b25","Type":"ContainerDied","Data":"1cde91bdbba3338478513ff2aefc100c01147272d9608289015318b826501de1"} Nov 03 02:10:16.241603 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.241538 4992 generic.go:356] "Generic (PLEG): container finished" podID="132fe9d0-a453-440b-946c-862a26f3c87f" containerID="3bad48d76aeeeb24511f90a97246c9d5ecc260931807e23c75da6e31b89eb3e0" exitCode=0 Nov 03 02:10:16.241743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.241680 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" event={"ID":"132fe9d0-a453-440b-946c-862a26f3c87f","Type":"ContainerDied","Data":"3bad48d76aeeeb24511f90a97246c9d5ecc260931807e23c75da6e31b89eb3e0"} Nov 03 02:10:16.243875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.243847 4992 generic.go:356] "Generic (PLEG): container finished" podID="fc96c80b-b323-46da-8c66-3c5c57762655" containerID="a1e0e349fa4b2252045e4a1e877e2325d6e59c82302729873f86892f70b38b97" exitCode=0 Nov 03 02:10:16.243997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.243947 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" event={"ID":"fc96c80b-b323-46da-8c66-3c5c57762655","Type":"ContainerDied","Data":"a1e0e349fa4b2252045e4a1e877e2325d6e59c82302729873f86892f70b38b97"} Nov 03 02:10:16.245650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.245629 4992 generic.go:356] "Generic (PLEG): container finished" podID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerID="7ca9f1e66a50d33a1942700c22179862908133c8661d7316442c1c361b33db7a" exitCode=0 Nov 03 02:10:16.245790 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.245685 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" event={"ID":"79c2e0d7-c418-4e94-98f4-874b193e917d","Type":"ContainerDied","Data":"7ca9f1e66a50d33a1942700c22179862908133c8661d7316442c1c361b33db7a"} Nov 03 02:10:16.336262 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.336230 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:10:16.392177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.392146 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 02:10:16.412987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.412963 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:10:16.415890 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.415869 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:10:16.455129 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455098 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455140 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455180 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455211 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455237 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455265 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455303 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.455349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455330 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455365 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455414 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455444 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455505 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455501 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455736 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.455785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455746 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.456136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.455798 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.456136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456075 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.456252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456142 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:16.456252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456207 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.456537 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456504 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.456762 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456743 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.456896 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.456879 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.457054 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457038 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.457180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457164 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.457291 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457276 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.457426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457409 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.457612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457596 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.457745 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457730 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.457868 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457855 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458146 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.457117 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458048 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "kube-api-access-l8zv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458167 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458176 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458181 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458197 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458279 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.458496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458415 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458507 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458697 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458745 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458829 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458857 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458888 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458917 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\") pod \"6a1c1012-898b-469f-87e1-79cf30f11b25\" (UID: \"6a1c1012-898b-469f-87e1-79cf30f11b25\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458961 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.458986 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459000 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\") pod \"79c2e0d7-c418-4e94-98f4-874b193e917d\" (UID: \"79c2e0d7-c418-4e94-98f4-874b193e917d\") " Nov 03 02:10:16.459065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459055 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459092 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\") pod \"132fe9d0-a453-440b-946c-862a26f3c87f\" (UID: \"132fe9d0-a453-440b-946c-862a26f3c87f\") " Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459161 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459176 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\") pod \"fc96c80b-b323-46da-8c66-3c5c57762655\" (UID: \"fc96c80b-b323-46da-8c66-3c5c57762655\") " Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459197 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459291 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "kube-api-access-cpkkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459489 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459665 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.459736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459709 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459923 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459974 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.459997 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132fe9d0-a453-440b-946c-862a26f3c87f-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460012 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460026 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc96c80b-b323-46da-8c66-3c5c57762655-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460041 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zv2\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-kube-api-access-l8zv2\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460056 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460071 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460086 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460100 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460114 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpkkj\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-kube-api-access-cpkkj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460128 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460142 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460163 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132fe9d0-a453-440b-946c-862a26f3c87f-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460178 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c2e0d7-c418-4e94-98f4-874b193e917d-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460192 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460207 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460220 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c1012-898b-469f-87e1-79cf30f11b25-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460234 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460255 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460269 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460283 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460501 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460764 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.460936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460796 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.461487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.460956 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.461487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.461083 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.461487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.461316 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:10:16.461893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.461867 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "kube-api-access-ztwlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.462033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.462013 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:16.462160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.462124 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token" pod "132fe9d0-a453-440b-946c-862a26f3c87f" (UID: "132fe9d0-a453-440b-946c-862a26f3c87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.462419 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.462389 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:16.463007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.462984 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token" pod "6a1c1012-898b-469f-87e1-79cf30f11b25" (UID: "6a1c1012-898b-469f-87e1-79cf30f11b25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.463088 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.463026 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:10:16.463136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.463106 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6" pod "fc96c80b-b323-46da-8c66-3c5c57762655" (UID: "fc96c80b-b323-46da-8c66-3c5c57762655"). InnerVolumeSpecName "kube-api-access-mx5f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.463457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.463441 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token" pod "79c2e0d7-c418-4e94-98f4-874b193e917d" (UID: "79c2e0d7-c418-4e94-98f4-874b193e917d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:10:16.561320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561224 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc96c80b-b323-46da-8c66-3c5c57762655-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561270 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5f6\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-kube-api-access-mx5f6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561290 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561304 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561318 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561332 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561345 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561361 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc96c80b-b323-46da-8c66-3c5c57762655-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561375 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561389 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561402 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132fe9d0-a453-440b-946c-862a26f3c87f-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561418 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c2e0d7-c418-4e94-98f4-874b193e917d-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561431 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztwlw\" (UniqueName: \"kubernetes.io/projected/132fe9d0-a453-440b-946c-862a26f3c87f-kube-api-access-ztwlw\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561448 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c1012-898b-469f-87e1-79cf30f11b25-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:16.561669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:16.561462 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc96c80b-b323-46da-8c66-3c5c57762655-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:10:17.253632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.253603 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" Nov 03 02:10:17.253632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.253623 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg" event={"ID":"6a1c1012-898b-469f-87e1-79cf30f11b25","Type":"ContainerDied","Data":"bca7702f6ae1fe5ebd70edaf962b66e1ec930819297ce835cf6a27aaa92980fc"} Nov 03 02:10:17.254169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.253691 4992 scope.go:122] "RemoveContainer" containerID="1cde91bdbba3338478513ff2aefc100c01147272d9608289015318b826501de1" Nov 03 02:10:17.255202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.255178 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" Nov 03 02:10:17.255321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.255184 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp" event={"ID":"132fe9d0-a453-440b-946c-862a26f3c87f","Type":"ContainerDied","Data":"2186cca9e18f773aa214b3e7e76ea7beee2486e0280de3a55f8bd8fb8a92f9c2"} Nov 03 02:10:17.257001 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.256984 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" Nov 03 02:10:17.257122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.257026 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66" event={"ID":"fc96c80b-b323-46da-8c66-3c5c57762655","Type":"ContainerDied","Data":"4295d2170d4119bee719acbc4f1acb3abbe4a235d227919dd026b6351d035b28"} Nov 03 02:10:17.258663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.258642 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" Nov 03 02:10:17.258769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.258667 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl" event={"ID":"79c2e0d7-c418-4e94-98f4-874b193e917d","Type":"ContainerDied","Data":"d6751037d7b862be61b8b81ad84ef3ea080db68ed7ce0b29d2efea6a188d7424"} Nov 03 02:10:17.264996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.264978 4992 scope.go:122] "RemoveContainer" containerID="3bad48d76aeeeb24511f90a97246c9d5ecc260931807e23c75da6e31b89eb3e0" Nov 03 02:10:17.273983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.273963 4992 scope.go:122] "RemoveContainer" containerID="a1e0e349fa4b2252045e4a1e877e2325d6e59c82302729873f86892f70b38b97" Nov 03 02:10:17.282524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.282497 4992 scope.go:122] "RemoveContainer" containerID="7ca9f1e66a50d33a1942700c22179862908133c8661d7316442c1c361b33db7a" Nov 03 02:10:17.425004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.424969 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:10:17.476981 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.476946 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-bmfxp"] Nov 03 02:10:17.542188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.542097 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:10:17.546073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.546040 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-8n9tl"] Nov 03 02:10:17.662128 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.662088 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 02:10:17.668738 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.668696 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-zfq66"] Nov 03 02:10:17.777983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.777943 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:10:17.781612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:17.781584 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-gkzmg"] Nov 03 02:10:18.598749 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:18.598713 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" path="/var/lib/kubelet/pods/132fe9d0-a453-440b-946c-862a26f3c87f/volumes" Nov 03 02:10:18.599307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:18.599290 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" path="/var/lib/kubelet/pods/6a1c1012-898b-469f-87e1-79cf30f11b25/volumes" Nov 03 02:10:18.599956 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:18.599937 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" path="/var/lib/kubelet/pods/79c2e0d7-c418-4e94-98f4-874b193e917d/volumes" Nov 03 02:10:18.600699 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:18.600684 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" path="/var/lib/kubelet/pods/fc96c80b-b323-46da-8c66-3c5c57762655/volumes" Nov 03 02:10:25.263217 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263175 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263714 4992 cpu_manager.go:414] "containerMap: removing container" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263729 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263752 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263757 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263777 4992 cpu_manager.go:414] "containerMap: removing container" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" Nov 03 02:10:25.263784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263783 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263795 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263802 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263818 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263823 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263840 4992 cpu_manager.go:414] "containerMap: removing container" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263848 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263863 4992 cpu_manager.go:414] "containerMap: removing container" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263867 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263954 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="7029f41c-6dc0-4dc1-9bfc-808b89943f33" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263962 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c2e0d7-c418-4e94-98f4-874b193e917d" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263972 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1c1012-898b-469f-87e1-79cf30f11b25" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263979 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc96c80b-b323-46da-8c66-3c5c57762655" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263985 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db16e50-75ce-46cb-ac39-01e9b20e4625" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263992 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5c61fe-a79d-449e-be65-56dae3e60e68" containerName="istio-proxy" Nov 03 02:10:25.264117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.263998 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="132fe9d0-a453-440b-946c-862a26f3c87f" containerName="istio-proxy" Nov 03 02:10:25.269194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.269165 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.324133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.323193 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:10:25.363887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.363853 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.364082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.363998 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbq78\" (UniqueName: \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.364127 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.364090 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.364127 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.364115 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.364217 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.364135 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.364296 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.364276 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.371382 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.371353 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-434b6a\"/\"kube-root-ca.crt\"" Nov 03 02:10:25.371507 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.371401 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-434b6a\"/\"default-dockercfg-fzflx\"" Nov 03 02:10:25.371507 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.371401 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-434b6a\"/\"openshift-service-ca.crt\"" Nov 03 02:10:25.374997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.374975 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-434b6a\"/\"auth-test-model-434b6a-kserve-self-signed-certs\"" Nov 03 02:10:25.465494 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465453 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.465732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465553 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbq78\" (UniqueName: \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.465732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465660 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.465732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465682 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.465732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465703 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.465966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465746 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.466025 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.465962 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.466089 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.466065 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.466126 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.466096 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.468159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.468138 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.468546 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.468525 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.529802 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.529716 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbq78\" (UniqueName: \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\") pod \"auth-test-model-434b6a-kserve-77bc876b8f-l8cfp\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.584167 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.584111 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:25.971765 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:10:25.971730 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd91334c_0053_4b79_a873_d7cd0321153a.slice/crio-538e0223432efba957c30804fd9e64bfee68d81060636dbd74a67994027bf86d WatchSource:0}: Error finding container 538e0223432efba957c30804fd9e64bfee68d81060636dbd74a67994027bf86d: Status 404 returned error can't find the container with id 538e0223432efba957c30804fd9e64bfee68d81060636dbd74a67994027bf86d Nov 03 02:10:25.973834 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:25.973807 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:10:26.318899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:26.318810 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" event={"ID":"dd91334c-0053-4b79-a873-d7cd0321153a","Type":"ContainerStarted","Data":"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754"} Nov 03 02:10:26.318899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:26.318855 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" event={"ID":"dd91334c-0053-4b79-a873-d7cd0321153a","Type":"ContainerStarted","Data":"538e0223432efba957c30804fd9e64bfee68d81060636dbd74a67994027bf86d"} Nov 03 02:10:26.439909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:26.439855 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" podStartSLOduration=1.439840847 podStartE2EDuration="1.439840847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:26.438650808 +0000 UTC m=+3162.017473395" watchObservedRunningTime="2026-11-03 02:10:26.439840847 +0000 UTC m=+3162.018663456" Nov 03 02:10:26.584830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:26.584741 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:32.152062 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:10:32.152019 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:10:35.584644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:35.584602 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:35.597886 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:35.597856 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:36.398900 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:36.398868 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:10:51.230203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.230162 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:10:51.235945 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.235922 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.289157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.289121 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:10:51.338150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.338114 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1c9b4e\"/\"auth-test-model-1c9b4e-kserve-self-signed-certs\"" Nov 03 02:10:51.338150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.338135 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1c9b4e\"/\"default-dockercfg-tcn2g\"" Nov 03 02:10:51.338150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.338149 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1c9b4e\"/\"kube-root-ca.crt\"" Nov 03 02:10:51.342106 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.342079 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1c9b4e\"/\"openshift-service-ca.crt\"" Nov 03 02:10:51.371692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.371659 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.371861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.371719 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.371861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.371792 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.371861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.371838 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.371861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.371856 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.372030 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.372009 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgbr\" (UniqueName: \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.473732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.473681 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.473943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.473767 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.473943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.473812 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.473943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.473877 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.473943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.473900 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.474211 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.474018 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgbr\" (UniqueName: \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.474360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.474307 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.474360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.474354 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.474552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.474395 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.476494 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.476473 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.476639 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.476538 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.537021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.536946 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgbr\" (UniqueName: \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\") pod \"auth-test-model-1c9b4e-kserve-8b746b69c-bt64r\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.549080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.548991 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:51.956329 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:10:51.956296 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd311a889_6fe4_4266_bc01_198ac2a8e3fc.slice/crio-f74e86613a2e0479a2524d263a1ec88daf5a371bf78c1e1ed02b3cc7f763e871 WatchSource:0}: Error finding container f74e86613a2e0479a2524d263a1ec88daf5a371bf78c1e1ed02b3cc7f763e871: Status 404 returned error can't find the container with id f74e86613a2e0479a2524d263a1ec88daf5a371bf78c1e1ed02b3cc7f763e871 Nov 03 02:10:51.958652 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:51.958625 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:10:52.492266 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:52.491401 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" event={"ID":"d311a889-6fe4-4266-bc01-198ac2a8e3fc","Type":"ContainerStarted","Data":"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b"} Nov 03 02:10:52.492266 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:52.491449 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" event={"ID":"d311a889-6fe4-4266-bc01-198ac2a8e3fc","Type":"ContainerStarted","Data":"f74e86613a2e0479a2524d263a1ec88daf5a371bf78c1e1ed02b3cc7f763e871"} Nov 03 02:10:52.549303 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:52.549265 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:10:52.613929 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:10:52.613880 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" podStartSLOduration=1.613866476 podStartE2EDuration="1.613866476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:52.612043135 +0000 UTC m=+3188.190865723" watchObservedRunningTime="2026-11-03 02:10:52.613866476 +0000 UTC m=+3188.192689066" Nov 03 02:11:01.549684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:01.549600 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:11:01.562825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:01.562793 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:11:02.152604 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:11:02.152548 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:11:02.561088 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:02.561052 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:11:32.155373 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:11:32.155324 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:11:33.689339 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:33.689303 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:11:33.689801 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:33.689502 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" podUID="17728b2d-56df-40b2-8842-4085dbd92a29" containerName="maas-api" containerID="cri-o://b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de" gracePeriod=30 Nov 03 02:11:33.992182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:33.992150 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:11:34.006151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006104 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\") pod \"17728b2d-56df-40b2-8842-4085dbd92a29\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " Nov 03 02:11:34.006381 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006202 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") pod \"17728b2d-56df-40b2-8842-4085dbd92a29\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " Nov 03 02:11:34.006381 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006296 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\" (UniqueName: \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\") pod \"17728b2d-56df-40b2-8842-4085dbd92a29\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " Nov 03 02:11:34.006508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006378 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\") pod \"17728b2d-56df-40b2-8842-4085dbd92a29\" (UID: \"17728b2d-56df-40b2-8842-4085dbd92a29\") " Nov 03 02:11:34.006698 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006650 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca" pod "17728b2d-56df-40b2-8842-4085dbd92a29" (UID: "17728b2d-56df-40b2-8842-4085dbd92a29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:11:34.006990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.006965 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17728b2d-56df-40b2-8842-4085dbd92a29-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:34.008776 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.008744 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs" pod "17728b2d-56df-40b2-8842-4085dbd92a29" (UID: "17728b2d-56df-40b2-8842-4085dbd92a29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:34.009066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.009046 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz" pod "17728b2d-56df-40b2-8842-4085dbd92a29" (UID: "17728b2d-56df-40b2-8842-4085dbd92a29"). InnerVolumeSpecName "kube-api-access-pg8sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:11:34.009137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.009053 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls" pod "17728b2d-56df-40b2-8842-4085dbd92a29" (UID: "17728b2d-56df-40b2-8842-4085dbd92a29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:11:34.122311 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.122218 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:34.122311 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.122269 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8sz\" (UniqueName: \"kubernetes.io/projected/17728b2d-56df-40b2-8842-4085dbd92a29-kube-api-access-pg8sz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:34.122311 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.122285 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17728b2d-56df-40b2-8842-4085dbd92a29-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:11:34.761281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.761245 4992 generic.go:356] "Generic (PLEG): container finished" podID="17728b2d-56df-40b2-8842-4085dbd92a29" containerID="b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de" exitCode=0 Nov 03 02:11:34.761761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.761297 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" event={"ID":"17728b2d-56df-40b2-8842-4085dbd92a29","Type":"ContainerDied","Data":"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de"} Nov 03 02:11:34.761761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.761306 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" Nov 03 02:11:34.761761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.761347 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l" event={"ID":"17728b2d-56df-40b2-8842-4085dbd92a29","Type":"ContainerDied","Data":"05001e78c4781f9b399dee31c1fc3deb7c7a17588fb1144144d9cee5c2f37d33"} Nov 03 02:11:34.761761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.761372 4992 scope.go:122] "RemoveContainer" containerID="b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de" Nov 03 02:11:34.771275 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.771251 4992 scope.go:122] "RemoveContainer" containerID="b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de" Nov 03 02:11:34.771616 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:11:34.771591 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de\": container with ID starting with b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de not found: ID does not exist" containerID="b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de" Nov 03 02:11:34.771705 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.771629 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de"} err="failed to get container status \"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de\": rpc error: code = NotFound desc = could not find container \"b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de\": container with ID starting with b5278572e138fcb5b4707b3cc04589f31bdc8ebf2580e6628449e9252712d0de not found: ID does not exist" Nov 03 02:11:34.932608 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.932545 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:11:34.949278 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:34.949235 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f7f61d-7899859fdf-k2t5l"] Nov 03 02:11:36.594451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:11:36.594402 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17728b2d-56df-40b2-8842-4085dbd92a29" path="/var/lib/kubelet/pods/17728b2d-56df-40b2-8842-4085dbd92a29/volumes" Nov 03 02:12:02.151639 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:12:02.151596 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:12:06.659072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.659031 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:12:06.659822 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.659805 4992 cpu_manager.go:414] "containerMap: removing container" podUID="17728b2d-56df-40b2-8842-4085dbd92a29" containerName="maas-api" Nov 03 02:12:06.659881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.659825 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17728b2d-56df-40b2-8842-4085dbd92a29" containerName="maas-api" Nov 03 02:12:06.659980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.659969 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="17728b2d-56df-40b2-8842-4085dbd92a29" containerName="maas-api" Nov 03 02:12:06.665002 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.664974 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707064 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707024 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707077 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707108 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707199 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707428 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707331 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.707611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.707587 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klk4w\" (UniqueName: \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.717940 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.717906 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:12:06.765384 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.765345 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"kube-root-ca.crt\"" Nov 03 02:12:06.765384 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.765373 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"auth-test-model-d9576a-kserve-self-signed-certs\"" Nov 03 02:12:06.765384 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.765361 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"default-dockercfg-g8sfg\"" Nov 03 02:12:06.765687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.765345 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9576a\"/\"openshift-service-ca.crt\"" Nov 03 02:12:06.809386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.809344 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810079 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klk4w\" (UniqueName: \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810182 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810213 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810238 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810267 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.810774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.810735 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.811536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.811448 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.812117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.811866 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.813164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.813136 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.814865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.814794 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.874232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.874190 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klk4w\" (UniqueName: \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\") pod \"auth-test-model-d9576a-kserve-655c5588f-ksclz\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:06.977692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:06.977658 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:07.358298 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:12:07.358261 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ab19aa_1701_4407_83b7_59ac03f1b64a.slice/crio-a6ae1d6cfecbecedc22dbe7c65999779ff76d73b9fc92da1a8a839f5d950fa30 WatchSource:0}: Error finding container a6ae1d6cfecbecedc22dbe7c65999779ff76d73b9fc92da1a8a839f5d950fa30: Status 404 returned error can't find the container with id a6ae1d6cfecbecedc22dbe7c65999779ff76d73b9fc92da1a8a839f5d950fa30 Nov 03 02:12:07.361201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:07.361175 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:12:07.975431 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:07.975388 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" event={"ID":"98ab19aa-1701-4407-83b7-59ac03f1b64a","Type":"ContainerStarted","Data":"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b"} Nov 03 02:12:07.975904 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:07.975445 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" event={"ID":"98ab19aa-1701-4407-83b7-59ac03f1b64a","Type":"ContainerStarted","Data":"a6ae1d6cfecbecedc22dbe7c65999779ff76d73b9fc92da1a8a839f5d950fa30"} Nov 03 02:12:07.978699 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:07.978684 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:08.096733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:08.096675 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" podStartSLOduration=2.096660636 podStartE2EDuration="2.096660636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:08.095286433 +0000 UTC m=+3263.674109020" watchObservedRunningTime="2026-11-03 02:12:08.096660636 +0000 UTC m=+3263.675483276" Nov 03 02:12:16.978313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:16.978272 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:16.991056 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:16.991014 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:17.050654 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:17.050622 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:12:32.152829 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:12:32.152787 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:12:45.064442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:45.064413 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:12:45.066633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:12:45.066603 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:13:02.152471 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:13:02.152431 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:13:32.151466 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:13:32.151423 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:13:56.453724 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.453625 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2"] Nov 03 02:13:56.457633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.457612 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.511124 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.511086 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2"] Nov 03 02:13:56.560072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.560036 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:13:56.560248 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.560127 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:13:56.578333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.578292 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjwn\" (UniqueName: \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.578333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.578340 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.578569 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.578478 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.680107 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.680064 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjwn\" (UniqueName: \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.680107 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.680115 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.680362 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.680184 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.680996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.680970 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.682679 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.682659 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.744731 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.744648 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjwn\" (UniqueName: \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\") pod \"maas-api-revoke-sub-43616389-zrdf2\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:56.771836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:56.771801 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:57.148787 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:13:57.148753 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913c53c5_6fb4_4fb0_898e_93313fd11d32.slice/crio-590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f WatchSource:0}: Error finding container 590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f: Status 404 returned error can't find the container with id 590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f Nov 03 02:13:57.151488 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:57.151459 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2"] Nov 03 02:13:57.653271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:57.653235 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" event={"ID":"913c53c5-6fb4-4fb0-898e-93313fd11d32","Type":"ContainerStarted","Data":"19c9f13931189548df979337ff21dec87333f67a7af1150694994aee2581a411"} Nov 03 02:13:57.653741 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:57.653299 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" event={"ID":"913c53c5-6fb4-4fb0-898e-93313fd11d32","Type":"ContainerStarted","Data":"590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f"} Nov 03 02:13:57.810170 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:57.810048 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" podStartSLOduration=1.810029425 podStartE2EDuration="1.810029425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:57.771988587 +0000 UTC m=+3373.350811173" watchObservedRunningTime="2026-11-03 02:13:57.810029425 +0000 UTC m=+3373.388852011" Nov 03 02:13:58.665819 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:58.665774 4992 generic.go:356] "Generic (PLEG): container finished" podID="913c53c5-6fb4-4fb0-898e-93313fd11d32" containerID="19c9f13931189548df979337ff21dec87333f67a7af1150694994aee2581a411" exitCode=0 Nov 03 02:13:58.666212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:58.665858 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" event={"ID":"913c53c5-6fb4-4fb0-898e-93313fd11d32","Type":"ContainerDied","Data":"19c9f13931189548df979337ff21dec87333f67a7af1150694994aee2581a411"} Nov 03 02:13:59.829270 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.829231 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:13:59.923247 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.922971 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\") pod \"913c53c5-6fb4-4fb0-898e-93313fd11d32\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " Nov 03 02:13:59.923417 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.923267 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\" (UniqueName: \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\") pod \"913c53c5-6fb4-4fb0-898e-93313fd11d32\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " Nov 03 02:13:59.923417 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.923370 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\") pod \"913c53c5-6fb4-4fb0-898e-93313fd11d32\" (UID: \"913c53c5-6fb4-4fb0-898e-93313fd11d32\") " Nov 03 02:13:59.923509 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.923470 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca" pod "913c53c5-6fb4-4fb0-898e-93313fd11d32" (UID: "913c53c5-6fb4-4fb0-898e-93313fd11d32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:13:59.923846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.923824 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:13:59.925430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.925406 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca" pod "913c53c5-6fb4-4fb0-898e-93313fd11d32" (UID: "913c53c5-6fb4-4fb0-898e-93313fd11d32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:13:59.925529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:13:59.925504 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn" pod "913c53c5-6fb4-4fb0-898e-93313fd11d32" (UID: "913c53c5-6fb4-4fb0-898e-93313fd11d32"). InnerVolumeSpecName "kube-api-access-pwjwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:00.025536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:00.025495 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjwn\" (UniqueName: \"kubernetes.io/projected/913c53c5-6fb4-4fb0-898e-93313fd11d32-kube-api-access-pwjwn\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:00.025536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:00.025532 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/913c53c5-6fb4-4fb0-898e-93313fd11d32-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:00.679887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:00.679847 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" event={"ID":"913c53c5-6fb4-4fb0-898e-93313fd11d32","Type":"ContainerDied","Data":"590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f"} Nov 03 02:14:00.680090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:00.679916 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590076497debf33a6574567070c0679001599a3e44f59923280c0fd62427e42f" Nov 03 02:14:00.680090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:00.679858 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2" Nov 03 02:14:02.151686 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:14:02.151649 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:14:07.184257 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.184218 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc"] Nov 03 02:14:07.184788 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.184770 4992 cpu_manager.go:414] "containerMap: removing container" podUID="913c53c5-6fb4-4fb0-898e-93313fd11d32" containerName="revoke-keys" Nov 03 02:14:07.184841 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.184788 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913c53c5-6fb4-4fb0-898e-93313fd11d32" containerName="revoke-keys" Nov 03 02:14:07.185018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.185005 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="913c53c5-6fb4-4fb0-898e-93313fd11d32" containerName="revoke-keys" Nov 03 02:14:07.188642 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.188619 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.214968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.214888 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqgq\" (UniqueName: \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.215180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.215081 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.215246 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.215199 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.242236 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.242197 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc"] Nov 03 02:14:07.291018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.290981 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:14:07.291203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.291049 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:14:07.316731 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.316687 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.316953 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.316786 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.316953 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.316892 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqgq\" (UniqueName: \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.317505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.317478 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.319522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.319497 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.379977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.379936 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqgq\" (UniqueName: \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\") pod \"maas-api-revoke-sub-0bd5f318-2kkjc\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.502270 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.502179 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:07.879767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:07.879726 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc"] Nov 03 02:14:07.880113 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:14:07.880088 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0874cb_1153_438a_a0b6_4c72d28e2ede.slice/crio-dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1 WatchSource:0}: Error finding container dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1: Status 404 returned error can't find the container with id dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1 Nov 03 02:14:08.730441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:08.730396 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" event={"ID":"ea0874cb-1153-438a-a0b6-4c72d28e2ede","Type":"ContainerStarted","Data":"f8d588ae114959c67b3d19d61d24b439fad2dcdd8427c3b2e15925f8f2f380b5"} Nov 03 02:14:08.730861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:08.730469 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" event={"ID":"ea0874cb-1153-438a-a0b6-4c72d28e2ede","Type":"ContainerStarted","Data":"dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1"} Nov 03 02:14:08.853167 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:08.853103 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" podStartSLOduration=1.853087501 podStartE2EDuration="1.853087501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:08.851021346 +0000 UTC m=+3384.429843935" watchObservedRunningTime="2026-11-03 02:14:08.853087501 +0000 UTC m=+3384.431910087" Nov 03 02:14:09.738080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:09.738043 4992 generic.go:356] "Generic (PLEG): container finished" podID="ea0874cb-1153-438a-a0b6-4c72d28e2ede" containerID="f8d588ae114959c67b3d19d61d24b439fad2dcdd8427c3b2e15925f8f2f380b5" exitCode=0 Nov 03 02:14:09.738080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:09.738080 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" event={"ID":"ea0874cb-1153-438a-a0b6-4c72d28e2ede","Type":"ContainerDied","Data":"f8d588ae114959c67b3d19d61d24b439fad2dcdd8427c3b2e15925f8f2f380b5"} Nov 03 02:14:10.896762 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.896722 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:10.962542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.962504 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\") pod \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " Nov 03 02:14:10.962747 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.962653 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\" (UniqueName: \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\") pod \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " Nov 03 02:14:10.962817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.962800 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\") pod \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\" (UID: \"ea0874cb-1153-438a-a0b6-4c72d28e2ede\") " Nov 03 02:14:10.963161 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.963130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca" pod "ea0874cb-1153-438a-a0b6-4c72d28e2ede" (UID: "ea0874cb-1153-438a-a0b6-4c72d28e2ede"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:10.963340 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.963321 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:10.964791 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.964765 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca" pod "ea0874cb-1153-438a-a0b6-4c72d28e2ede" (UID: "ea0874cb-1153-438a-a0b6-4c72d28e2ede"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:10.964869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:10.964837 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq" pod "ea0874cb-1153-438a-a0b6-4c72d28e2ede" (UID: "ea0874cb-1153-438a-a0b6-4c72d28e2ede"). InnerVolumeSpecName "kube-api-access-tsqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:11.064872 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:11.064785 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0874cb-1153-438a-a0b6-4c72d28e2ede-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:11.064872 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:11.064821 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqgq\" (UniqueName: \"kubernetes.io/projected/ea0874cb-1153-438a-a0b6-4c72d28e2ede-kube-api-access-tsqgq\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:11.751352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:11.751317 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" Nov 03 02:14:11.751536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:11.751350 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc" event={"ID":"ea0874cb-1153-438a-a0b6-4c72d28e2ede","Type":"ContainerDied","Data":"dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1"} Nov 03 02:14:11.751536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:11.751403 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6ae7ef9824f1edcd0a259e5f809c0840f6ef2e1c3ae379cdb5b60817d535b1" Nov 03 02:14:22.535855 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:22.535811 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:14:32.151539 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:14:32.151500 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:14:37.556942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.556872 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" podUID="dd91334c-0053-4b79-a873-d7cd0321153a" containerName="main" containerID="cri-o://d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754" gracePeriod=45 Nov 03 02:14:37.831085 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.831057 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:14:37.884446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884407 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.884446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884449 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.884744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884517 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.884744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884615 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.884844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884752 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:37.884844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884783 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\" (UniqueName: \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.884844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884802 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:37.885010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884907 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\") pod \"dd91334c-0053-4b79-a873-d7cd0321153a\" (UID: \"dd91334c-0053-4b79-a873-d7cd0321153a\") " Nov 03 02:14:37.885010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.884931 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:37.885468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.885434 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.885468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.885460 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.885688 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.885477 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.887315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.887284 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "kube-api-access-fbq78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:37.887508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.887483 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:14:37.887508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.887493 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs" pod "dd91334c-0053-4b79-a873-d7cd0321153a" (UID: "dd91334c-0053-4b79-a873-d7cd0321153a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:37.926422 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.926333 4992 generic.go:356] "Generic (PLEG): container finished" podID="dd91334c-0053-4b79-a873-d7cd0321153a" containerID="d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754" exitCode=0 Nov 03 02:14:37.926422 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.926377 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" Nov 03 02:14:37.926665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.926421 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" event={"ID":"dd91334c-0053-4b79-a873-d7cd0321153a","Type":"ContainerDied","Data":"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754"} Nov 03 02:14:37.926665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.926469 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp" event={"ID":"dd91334c-0053-4b79-a873-d7cd0321153a","Type":"ContainerDied","Data":"538e0223432efba957c30804fd9e64bfee68d81060636dbd74a67994027bf86d"} Nov 03 02:14:37.926665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.926489 4992 scope.go:122] "RemoveContainer" containerID="d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754" Nov 03 02:14:37.936978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.936957 4992 scope.go:122] "RemoveContainer" containerID="d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754" Nov 03 02:14:37.937295 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:14:37.937272 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754\": container with ID starting with d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754 not found: ID does not exist" containerID="d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754" Nov 03 02:14:37.937347 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.937308 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754"} err="failed to get container status \"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754\": rpc error: code = NotFound desc = could not find container \"d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754\": container with ID starting with d534ceabd2933c8e91db350bc82c279d17cc3ff5ffdb5ec1d0df58a8226a2754 not found: ID does not exist" Nov 03 02:14:37.986955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.986913 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd91334c-0053-4b79-a873-d7cd0321153a-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.986955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.986951 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbq78\" (UniqueName: \"kubernetes.io/projected/dd91334c-0053-4b79-a873-d7cd0321153a-kube-api-access-fbq78\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:37.986955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:37.986963 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd91334c-0053-4b79-a873-d7cd0321153a-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:38.069130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:38.069076 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:14:38.075242 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:38.075201 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-434b6a/auth-test-model-434b6a-kserve-77bc876b8f-l8cfp"] Nov 03 02:14:38.595454 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:38.595416 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd91334c-0053-4b79-a873-d7cd0321153a" path="/var/lib/kubelet/pods/dd91334c-0053-4b79-a873-d7cd0321153a/volumes" Nov 03 02:14:44.180948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.180905 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb"] Nov 03 02:14:44.181430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181416 4992 cpu_manager.go:414] "containerMap: removing container" podUID="dd91334c-0053-4b79-a873-d7cd0321153a" containerName="main" Nov 03 02:14:44.181474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181431 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd91334c-0053-4b79-a873-d7cd0321153a" containerName="main" Nov 03 02:14:44.181474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181457 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ea0874cb-1153-438a-a0b6-4c72d28e2ede" containerName="revoke-keys" Nov 03 02:14:44.181474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181462 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0874cb-1153-438a-a0b6-4c72d28e2ede" containerName="revoke-keys" Nov 03 02:14:44.181598 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181587 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd91334c-0053-4b79-a873-d7cd0321153a" containerName="main" Nov 03 02:14:44.181637 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.181602 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0874cb-1153-438a-a0b6-4c72d28e2ede" containerName="revoke-keys" Nov 03 02:14:44.186399 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.186377 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.238146 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.238102 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb"] Nov 03 02:14:44.273181 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.273123 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.273405 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.273295 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fp7\" (UniqueName: \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.273405 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.273329 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.286888 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.286855 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:14:44.287074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.286962 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:14:44.374498 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.374461 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.374711 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.374542 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fp7\" (UniqueName: \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.374711 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.374575 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.375283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.375258 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.377115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.377089 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.438152 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.438052 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fp7\" (UniqueName: \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\") pod \"maas-api-revoke-sub-ee74161d-c7ntb\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.500140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.500101 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:44.881615 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:14:44.881583 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e253e5_fbc1_40dc_877c_58d37ab03f29.slice/crio-1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141 WatchSource:0}: Error finding container 1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141: Status 404 returned error can't find the container with id 1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141 Nov 03 02:14:44.884082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.884052 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb"] Nov 03 02:14:44.983548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:44.983512 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" event={"ID":"29e253e5-fbc1-40dc-877c-58d37ab03f29","Type":"ContainerStarted","Data":"1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141"} Nov 03 02:14:45.991388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:45.991353 4992 generic.go:356] "Generic (PLEG): container finished" podID="29e253e5-fbc1-40dc-877c-58d37ab03f29" containerID="6f5c7673896bd9a9ccdf3960cae3d8e66bf5ddcd2d44fa28da8bf64b094c08bd" exitCode=0 Nov 03 02:14:45.991862 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:45.991437 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" event={"ID":"29e253e5-fbc1-40dc-877c-58d37ab03f29","Type":"ContainerDied","Data":"6f5c7673896bd9a9ccdf3960cae3d8e66bf5ddcd2d44fa28da8bf64b094c08bd"} Nov 03 02:14:47.161627 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.161592 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:47.306448 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.306345 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\" (UniqueName: \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\") pod \"29e253e5-fbc1-40dc-877c-58d37ab03f29\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " Nov 03 02:14:47.306448 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.306416 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\") pod \"29e253e5-fbc1-40dc-877c-58d37ab03f29\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " Nov 03 02:14:47.306736 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.306482 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\") pod \"29e253e5-fbc1-40dc-877c-58d37ab03f29\" (UID: \"29e253e5-fbc1-40dc-877c-58d37ab03f29\") " Nov 03 02:14:47.306958 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.306929 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca" pod "29e253e5-fbc1-40dc-877c-58d37ab03f29" (UID: "29e253e5-fbc1-40dc-877c-58d37ab03f29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:47.307105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.307084 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:47.309336 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.309305 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7" pod "29e253e5-fbc1-40dc-877c-58d37ab03f29" (UID: "29e253e5-fbc1-40dc-877c-58d37ab03f29"). InnerVolumeSpecName "kube-api-access-77fp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:47.309712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.309687 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca" pod "29e253e5-fbc1-40dc-877c-58d37ab03f29" (UID: "29e253e5-fbc1-40dc-877c-58d37ab03f29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:47.407905 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.407865 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fp7\" (UniqueName: \"kubernetes.io/projected/29e253e5-fbc1-40dc-877c-58d37ab03f29-kube-api-access-77fp7\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:47.407905 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:47.407901 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e253e5-fbc1-40dc-877c-58d37ab03f29-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:48.004915 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:48.004883 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" Nov 03 02:14:48.004915 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:48.004908 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb" event={"ID":"29e253e5-fbc1-40dc-877c-58d37ab03f29","Type":"ContainerDied","Data":"1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141"} Nov 03 02:14:48.005137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:48.004957 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1052a1d1e6b6344746be0b10e97a18b68c1488069e598e773675ad9953cca141" Nov 03 02:14:54.913199 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.913159 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654"] Nov 03 02:14:54.913912 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.913889 4992 cpu_manager.go:414] "containerMap: removing container" podUID="29e253e5-fbc1-40dc-877c-58d37ab03f29" containerName="revoke-keys" Nov 03 02:14:54.914011 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.913913 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e253e5-fbc1-40dc-877c-58d37ab03f29" containerName="revoke-keys" Nov 03 02:14:54.914115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.914101 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e253e5-fbc1-40dc-877c-58d37ab03f29" containerName="revoke-keys" Nov 03 02:14:54.917640 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.917612 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:54.971178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:54.971145 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654"] Nov 03 02:14:55.003823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.003784 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxbm\" (UniqueName: \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.004033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.003930 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.004033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.003985 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.019416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.019381 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:14:55.019609 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.019478 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:14:55.105056 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.105012 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxbm\" (UniqueName: \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.105210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.105136 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.105210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.105181 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.105956 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.105932 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.107637 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.107613 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.168735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.168649 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxbm\" (UniqueName: \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\") pod \"maas-api-revoke-sub-7bb8a9ad-kg654\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.230854 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.230821 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:55.608069 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:14:55.608030 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce8f1d0_7598_4ee8_a417_1280495c5086.slice/crio-8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880 WatchSource:0}: Error finding container 8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880: Status 404 returned error can't find the container with id 8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880 Nov 03 02:14:55.610647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:55.610546 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654"] Nov 03 02:14:56.057851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:56.057813 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" event={"ID":"4ce8f1d0-7598-4ee8-a417-1280495c5086","Type":"ContainerStarted","Data":"2fbadcbc6b4a677f7a1a435e148a4047aa2d9f15fe0264b52bf68a5bb3235509"} Nov 03 02:14:56.058249 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:56.057880 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" event={"ID":"4ce8f1d0-7598-4ee8-a417-1280495c5086","Type":"ContainerStarted","Data":"8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880"} Nov 03 02:14:56.184749 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:56.184681 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" podStartSLOduration=2.184664632 podStartE2EDuration="2.184664632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:56.18223656 +0000 UTC m=+3431.761059147" watchObservedRunningTime="2026-11-03 02:14:56.184664632 +0000 UTC m=+3431.763487218" Nov 03 02:14:57.065456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:57.065419 4992 generic.go:356] "Generic (PLEG): container finished" podID="4ce8f1d0-7598-4ee8-a417-1280495c5086" containerID="2fbadcbc6b4a677f7a1a435e148a4047aa2d9f15fe0264b52bf68a5bb3235509" exitCode=0 Nov 03 02:14:57.065899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:57.065514 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" event={"ID":"4ce8f1d0-7598-4ee8-a417-1280495c5086","Type":"ContainerDied","Data":"2fbadcbc6b4a677f7a1a435e148a4047aa2d9f15fe0264b52bf68a5bb3235509"} Nov 03 02:14:58.250889 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.250859 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:58.345429 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.345336 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\") pod \"4ce8f1d0-7598-4ee8-a417-1280495c5086\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " Nov 03 02:14:58.345622 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.345442 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\" (UniqueName: \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\") pod \"4ce8f1d0-7598-4ee8-a417-1280495c5086\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " Nov 03 02:14:58.345622 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.345542 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\") pod \"4ce8f1d0-7598-4ee8-a417-1280495c5086\" (UID: \"4ce8f1d0-7598-4ee8-a417-1280495c5086\") " Nov 03 02:14:58.345914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.345890 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca" pod "4ce8f1d0-7598-4ee8-a417-1280495c5086" (UID: "4ce8f1d0-7598-4ee8-a417-1280495c5086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:14:58.346038 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.346022 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:58.347671 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.347641 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca" pod "4ce8f1d0-7598-4ee8-a417-1280495c5086" (UID: "4ce8f1d0-7598-4ee8-a417-1280495c5086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:14:58.347785 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.347674 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm" pod "4ce8f1d0-7598-4ee8-a417-1280495c5086" (UID: "4ce8f1d0-7598-4ee8-a417-1280495c5086"). InnerVolumeSpecName "kube-api-access-9lxbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:14:58.447185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.447140 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ce8f1d0-7598-4ee8-a417-1280495c5086-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:58.447185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:58.447180 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxbm\" (UniqueName: \"kubernetes.io/projected/4ce8f1d0-7598-4ee8-a417-1280495c5086-kube-api-access-9lxbm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:14:59.078299 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:59.078264 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" Nov 03 02:14:59.078485 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:59.078293 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654" event={"ID":"4ce8f1d0-7598-4ee8-a417-1280495c5086","Type":"ContainerDied","Data":"8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880"} Nov 03 02:14:59.078485 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:14:59.078343 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d9cc15915dbbda475922a0aca809deaed402af9e3863ac55240559c5df82880" Nov 03 02:15:00.165202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.165155 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:00.165927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.165902 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4ce8f1d0-7598-4ee8-a417-1280495c5086" containerName="revoke-keys" Nov 03 02:15:00.165927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.165928 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ce8f1d0-7598-4ee8-a417-1280495c5086" containerName="revoke-keys" Nov 03 02:15:00.166103 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.166093 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ce8f1d0-7598-4ee8-a417-1280495c5086" containerName="revoke-keys" Nov 03 02:15:00.169944 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.169921 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:00.170049 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.170027 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.174028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.174000 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:00.174136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.174106 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.178043 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.177860 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.217333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.217295 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:00.221395 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.221364 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:00.221589 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.221510 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.225796 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.225762 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:00.225938 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.225878 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.229939 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.229875 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267590 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfg7s\" (UniqueName: \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267661 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267702 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw86\" (UniqueName: \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267744 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267798 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267869 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmxq\" (UniqueName: \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.267983 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmct\" (UniqueName: \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.268033 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.268132 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.268181 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9h6m\" (UniqueName: \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.268242 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.268303 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tcbm\" (UniqueName: \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.272759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.269054 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:00.273368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.272978 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:00.322510 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.322473 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:15:00.327106 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.327071 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:00.332033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.332000 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:00.367930 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.367886 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:00.374252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.374216 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:00.374997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.374968 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmct\" (UniqueName: \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.375077 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375017 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.375122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375078 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.375154 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375121 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9h6m\" (UniqueName: \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.375199 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375183 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.375264 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375250 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcbm\" (UniqueName: \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.375361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375346 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfg7s\" (UniqueName: \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.375427 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375395 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.375427 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375420 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw86\" (UniqueName: \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.375710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375444 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.375710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375478 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.375710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.375549 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmxq\" (UniqueName: \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.376929 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.376901 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.377644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.377619 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.378286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.378264 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.378374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.378351 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.378523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.378428 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.378741 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.378716 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.442263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.442177 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmct\" (UniqueName: \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.442263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.442211 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9h6m\" (UniqueName: \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\") pod \"maas-api-key-cleanup-29894535-ncjsz\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.442263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.442179 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfg7s\" (UniqueName: \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.442263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.442211 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmxq\" (UniqueName: \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.442635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.442264 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcbm\" (UniqueName: \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.458795 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.458757 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw86\" (UniqueName: \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.485519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.485484 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:00.493080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.493055 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:00.502292 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.502258 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:00.538994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.538700 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:00.570126 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.569838 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:00.575048 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.573803 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:00.881280 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:00.881241 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb09b4c_0c69_41d7_a942_88f0db86e2b4.slice/crio-a2f3f5e10a9c675d937c29c4e4d575ec601401593a529c90551b533a684e64c4 WatchSource:0}: Error finding container a2f3f5e10a9c675d937c29c4e4d575ec601401593a529c90551b533a684e64c4: Status 404 returned error can't find the container with id a2f3f5e10a9c675d937c29c4e4d575ec601401593a529c90551b533a684e64c4 Nov 03 02:15:00.883859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.883838 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:00.929124 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:00.929078 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5932dba_f1a9_49f7_96a4_15ce53a18897.slice/crio-c366733bedb86bd1e56a32a7f8728df3f3158b317dab8d65ba09faffd0cfc3c4 WatchSource:0}: Error finding container c366733bedb86bd1e56a32a7f8728df3f3158b317dab8d65ba09faffd0cfc3c4: Status 404 returned error can't find the container with id c366733bedb86bd1e56a32a7f8728df3f3158b317dab8d65ba09faffd0cfc3c4 Nov 03 02:15:00.931801 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:00.931772 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613e96cb_0d92_408a_a7c8_19590d1d3761.slice/crio-40e6e4cdb9a76e317f27dc5f787142660e1eabb606d057350ae6d8efe05b9993 WatchSource:0}: Error finding container 40e6e4cdb9a76e317f27dc5f787142660e1eabb606d057350ae6d8efe05b9993: Status 404 returned error can't find the container with id 40e6e4cdb9a76e317f27dc5f787142660e1eabb606d057350ae6d8efe05b9993 Nov 03 02:15:00.932017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.931981 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:00.935377 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.935350 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:00.985067 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:00.985031 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf910323a_f6d3_4c46_889c_228b72c58256.slice/crio-c6b2f8471484147001e21c84d25958b6d8f5cee565c227c840d885e8b20cbf27 WatchSource:0}: Error finding container c6b2f8471484147001e21c84d25958b6d8f5cee565c227c840d885e8b20cbf27: Status 404 returned error can't find the container with id c6b2f8471484147001e21c84d25958b6d8f5cee565c227c840d885e8b20cbf27 Nov 03 02:15:00.985792 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.985708 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:00.987711 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:00.987682 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6840063a_82fe_41ae_b1a7_667ab2e7abea.slice/crio-20d5025c78e5a3f37916baabfb2e8456b2d4bcfb6af202653179ed875bda0e1e WatchSource:0}: Error finding container 20d5025c78e5a3f37916baabfb2e8456b2d4bcfb6af202653179ed875bda0e1e: Status 404 returned error can't find the container with id 20d5025c78e5a3f37916baabfb2e8456b2d4bcfb6af202653179ed875bda0e1e Nov 03 02:15:00.990879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:00.990857 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:01.020687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.020657 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:01.024898 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:01.024865 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc669af9_abc5_4e22_abae_b5e37ef7beab.slice/crio-6744ae8aa81ce9ad9596ab6f5e834e86c4b1b50ae20ccb8764a305aa576b75dd WatchSource:0}: Error finding container 6744ae8aa81ce9ad9596ab6f5e834e86c4b1b50ae20ccb8764a305aa576b75dd: Status 404 returned error can't find the container with id 6744ae8aa81ce9ad9596ab6f5e834e86c4b1b50ae20ccb8764a305aa576b75dd Nov 03 02:15:01.098777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.098734 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerStarted","Data":"a2f3f5e10a9c675d937c29c4e4d575ec601401593a529c90551b533a684e64c4"} Nov 03 02:15:01.101876 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.101835 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerStarted","Data":"6744ae8aa81ce9ad9596ab6f5e834e86c4b1b50ae20ccb8764a305aa576b75dd"} Nov 03 02:15:01.103427 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.103393 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerStarted","Data":"20d5025c78e5a3f37916baabfb2e8456b2d4bcfb6af202653179ed875bda0e1e"} Nov 03 02:15:01.104847 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.104818 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerStarted","Data":"c366733bedb86bd1e56a32a7f8728df3f3158b317dab8d65ba09faffd0cfc3c4"} Nov 03 02:15:01.106085 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.106062 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerStarted","Data":"40e6e4cdb9a76e317f27dc5f787142660e1eabb606d057350ae6d8efe05b9993"} Nov 03 02:15:01.107334 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:01.107310 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerStarted","Data":"c6b2f8471484147001e21c84d25958b6d8f5cee565c227c840d885e8b20cbf27"} Nov 03 02:15:02.116090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.115990 4992 generic.go:356] "Generic (PLEG): container finished" podID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerID="2c40a6e07d7244b241013199b071dfbc265642c4796947aee6388adac3c60cf1" exitCode=60 Nov 03 02:15:02.116090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.116072 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerDied","Data":"2c40a6e07d7244b241013199b071dfbc265642c4796947aee6388adac3c60cf1"} Nov 03 02:15:02.116548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.116380 4992 scope.go:122] "RemoveContainer" containerID="2c40a6e07d7244b241013199b071dfbc265642c4796947aee6388adac3c60cf1" Nov 03 02:15:02.117810 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.117781 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerStarted","Data":"76b7b1ee8b456b430159812ed6575e13a73261143019f320356908d2b9d16752"} Nov 03 02:15:02.119524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.119438 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerStarted","Data":"a05bc1af5257335b59a9dafbaa28fc55d9876d06f8e760db2cae21e7c1582b20"} Nov 03 02:15:02.121109 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.121087 4992 generic.go:356] "Generic (PLEG): container finished" podID="f910323a-f6d3-4c46-889c-228b72c58256" containerID="36925cc294dbf587704544cb0b64f4b4d1b32a82dabb782471a9341ab9cf2d09" exitCode=60 Nov 03 02:15:02.121216 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.121175 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerDied","Data":"36925cc294dbf587704544cb0b64f4b4d1b32a82dabb782471a9341ab9cf2d09"} Nov 03 02:15:02.121351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.121333 4992 scope.go:122] "RemoveContainer" containerID="36925cc294dbf587704544cb0b64f4b4d1b32a82dabb782471a9341ab9cf2d09" Nov 03 02:15:02.122882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.122854 4992 generic.go:356] "Generic (PLEG): container finished" podID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerID="19dc4d082b96ff4bf081fcd298a7dcf4aa9597580c12d41c1900f8f65d368dd7" exitCode=60 Nov 03 02:15:02.122991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.122901 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerDied","Data":"19dc4d082b96ff4bf081fcd298a7dcf4aa9597580c12d41c1900f8f65d368dd7"} Nov 03 02:15:02.123148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.123130 4992 scope.go:122] "RemoveContainer" containerID="19dc4d082b96ff4bf081fcd298a7dcf4aa9597580c12d41c1900f8f65d368dd7" Nov 03 02:15:02.124774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.124742 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerStarted","Data":"069fa289606c0f03d6c90cbfb2d07b03a2a420a0beaedda7800a9d23c87e3620"} Nov 03 02:15:02.158170 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:02.158134 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:15:02.474198 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.474139 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" podStartSLOduration=2.474119303 podStartE2EDuration="2.474119303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.473024707 +0000 UTC m=+3438.051847297" watchObservedRunningTime="2026-11-03 02:15:02.474119303 +0000 UTC m=+3438.052941890" Nov 03 02:15:02.596497 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.594853 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" podStartSLOduration=2.594833432 podStartE2EDuration="2.594833432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.591762422 +0000 UTC m=+3438.170585009" watchObservedRunningTime="2026-11-03 02:15:02.594833432 +0000 UTC m=+3438.173656021" Nov 03 02:15:02.829059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:02.829003 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" podStartSLOduration=2.828988233 podStartE2EDuration="2.828988233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.828713321 +0000 UTC m=+3438.407535909" watchObservedRunningTime="2026-11-03 02:15:02.828988233 +0000 UTC m=+3438.407810820" Nov 03 02:15:03.132397 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.132361 4992 generic.go:356] "Generic (PLEG): container finished" podID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerID="a05bc1af5257335b59a9dafbaa28fc55d9876d06f8e760db2cae21e7c1582b20" exitCode=60 Nov 03 02:15:03.132893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.132407 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerDied","Data":"a05bc1af5257335b59a9dafbaa28fc55d9876d06f8e760db2cae21e7c1582b20"} Nov 03 02:15:03.132893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.132763 4992 scope.go:122] "RemoveContainer" containerID="a05bc1af5257335b59a9dafbaa28fc55d9876d06f8e760db2cae21e7c1582b20" Nov 03 02:15:03.134902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.134485 4992 generic.go:356] "Generic (PLEG): container finished" podID="f910323a-f6d3-4c46-889c-228b72c58256" containerID="5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b" exitCode=60 Nov 03 02:15:03.134902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.134541 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerDied","Data":"5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b"} Nov 03 02:15:03.134902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.134618 4992 scope.go:122] "RemoveContainer" containerID="36925cc294dbf587704544cb0b64f4b4d1b32a82dabb782471a9341ab9cf2d09" Nov 03 02:15:03.134902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.134855 4992 scope.go:122] "RemoveContainer" containerID="5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b" Nov 03 02:15:03.135154 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:03.135128 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq_odh-ai-gateway-infra(f910323a-f6d3-4c46-889c-228b72c58256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" podUID="f910323a-f6d3-4c46-889c-228b72c58256" Nov 03 02:15:03.137063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.137038 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerStarted","Data":"42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41"} Nov 03 02:15:03.138767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.138746 4992 generic.go:356] "Generic (PLEG): container finished" podID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerID="069fa289606c0f03d6c90cbfb2d07b03a2a420a0beaedda7800a9d23c87e3620" exitCode=60 Nov 03 02:15:03.138874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.138830 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerDied","Data":"069fa289606c0f03d6c90cbfb2d07b03a2a420a0beaedda7800a9d23c87e3620"} Nov 03 02:15:03.139062 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.139046 4992 scope.go:122] "RemoveContainer" containerID="069fa289606c0f03d6c90cbfb2d07b03a2a420a0beaedda7800a9d23c87e3620" Nov 03 02:15:03.141239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.141213 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerStarted","Data":"061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323"} Nov 03 02:15:03.143102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.143073 4992 generic.go:356] "Generic (PLEG): container finished" podID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerID="76b7b1ee8b456b430159812ed6575e13a73261143019f320356908d2b9d16752" exitCode=60 Nov 03 02:15:03.143227 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.143127 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerDied","Data":"76b7b1ee8b456b430159812ed6575e13a73261143019f320356908d2b9d16752"} Nov 03 02:15:03.143363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.143345 4992 scope.go:122] "RemoveContainer" containerID="76b7b1ee8b456b430159812ed6575e13a73261143019f320356908d2b9d16752" Nov 03 02:15:03.491742 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.491689 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" podStartSLOduration=3.491674335 podStartE2EDuration="3.491674335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.491051296 +0000 UTC m=+3439.069873884" watchObservedRunningTime="2026-11-03 02:15:03.491674335 +0000 UTC m=+3439.070496921" Nov 03 02:15:03.853046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:03.852990 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" podStartSLOduration=3.852973413 podStartE2EDuration="3.852973413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:03.850318114 +0000 UTC m=+3439.429140706" watchObservedRunningTime="2026-11-03 02:15:03.852973413 +0000 UTC m=+3439.431796025" Nov 03 02:15:04.152073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.151979 4992 generic.go:356] "Generic (PLEG): container finished" podID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerID="42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41" exitCode=60 Nov 03 02:15:04.152073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.152036 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerDied","Data":"42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41"} Nov 03 02:15:04.152643 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.152106 4992 scope.go:122] "RemoveContainer" containerID="19dc4d082b96ff4bf081fcd298a7dcf4aa9597580c12d41c1900f8f65d368dd7" Nov 03 02:15:04.152643 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.152405 4992 scope.go:122] "RemoveContainer" containerID="42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41" Nov 03 02:15:04.152823 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:04.152793 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm_odh-ai-gateway-infra(eeb09b4c-0c69-41d7-a942-88f0db86e2b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" Nov 03 02:15:04.154938 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.154911 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerStarted","Data":"dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87"} Nov 03 02:15:04.156895 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.156873 4992 generic.go:356] "Generic (PLEG): container finished" podID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerID="061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323" exitCode=60 Nov 03 02:15:04.157005 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.156954 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerDied","Data":"061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323"} Nov 03 02:15:04.157291 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.157268 4992 scope.go:122] "RemoveContainer" containerID="061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323" Nov 03 02:15:04.157642 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:04.157620 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449_odh-ai-gateway-infra(6840063a-82fe-41ae-b1a7-667ab2e7abea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" Nov 03 02:15:04.158953 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.158924 4992 generic.go:356] "Generic (PLEG): container finished" podID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerID="57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1" exitCode=60 Nov 03 02:15:04.159066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.158959 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerDied","Data":"57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1"} Nov 03 02:15:04.159296 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.159278 4992 scope.go:122] "RemoveContainer" containerID="57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1" Nov 03 02:15:04.159641 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:04.159614 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv_odh-ai-gateway-infra(b5932dba-f1a9-49f7-96a4-15ce53a18897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" Nov 03 02:15:04.161019 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.160996 4992 generic.go:356] "Generic (PLEG): container finished" podID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerID="6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed" exitCode=60 Nov 03 02:15:04.161127 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.161076 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerDied","Data":"6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed"} Nov 03 02:15:04.161376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.161358 4992 scope.go:122] "RemoveContainer" containerID="6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed" Nov 03 02:15:04.161690 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:04.161655 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6_odh-ai-gateway-infra(613e96cb-0d92-408a-a7c8-19590d1d3761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" Nov 03 02:15:04.163243 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.163221 4992 scope.go:122] "RemoveContainer" containerID="5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b" Nov 03 02:15:04.163591 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:04.163532 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq_odh-ai-gateway-infra(f910323a-f6d3-4c46-889c-228b72c58256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" podUID="f910323a-f6d3-4c46-889c-228b72c58256" Nov 03 02:15:04.170034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.170009 4992 scope.go:122] "RemoveContainer" containerID="2c40a6e07d7244b241013199b071dfbc265642c4796947aee6388adac3c60cf1" Nov 03 02:15:04.261005 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.260978 4992 scope.go:122] "RemoveContainer" containerID="76b7b1ee8b456b430159812ed6575e13a73261143019f320356908d2b9d16752" Nov 03 02:15:04.276342 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:04.276310 4992 scope.go:122] "RemoveContainer" containerID="a05bc1af5257335b59a9dafbaa28fc55d9876d06f8e760db2cae21e7c1582b20" Nov 03 02:15:05.172596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.172543 4992 scope.go:122] "RemoveContainer" containerID="42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41" Nov 03 02:15:05.173094 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:05.172903 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm_odh-ai-gateway-infra(eeb09b4c-0c69-41d7-a942-88f0db86e2b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" Nov 03 02:15:05.174187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.174156 4992 generic.go:356] "Generic (PLEG): container finished" podID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerID="dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87" exitCode=60 Nov 03 02:15:05.174290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.174243 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerDied","Data":"dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87"} Nov 03 02:15:05.174355 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.174314 4992 scope.go:122] "RemoveContainer" containerID="069fa289606c0f03d6c90cbfb2d07b03a2a420a0beaedda7800a9d23c87e3620" Nov 03 02:15:05.174505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.174485 4992 scope.go:122] "RemoveContainer" containerID="dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87" Nov 03 02:15:05.174849 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:05.174821 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-ncjsz_odh-ai-gateway-infra(cc669af9-abc5-4e22-abae-b5e37ef7beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" Nov 03 02:15:05.176233 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.176215 4992 scope.go:122] "RemoveContainer" containerID="061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323" Nov 03 02:15:05.176483 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:05.176464 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449_odh-ai-gateway-infra(6840063a-82fe-41ae-b1a7-667ab2e7abea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" Nov 03 02:15:05.177922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.177901 4992 scope.go:122] "RemoveContainer" containerID="57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1" Nov 03 02:15:05.178152 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:05.178132 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv_odh-ai-gateway-infra(b5932dba-f1a9-49f7-96a4-15ce53a18897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" Nov 03 02:15:05.179320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:05.179302 4992 scope.go:122] "RemoveContainer" containerID="6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed" Nov 03 02:15:05.179505 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:05.179486 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6_odh-ai-gateway-infra(613e96cb-0d92-408a-a7c8-19590d1d3761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" Nov 03 02:15:06.188210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:06.188180 4992 scope.go:122] "RemoveContainer" containerID="dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87" Nov 03 02:15:06.188719 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:06.188553 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-ncjsz_odh-ai-gateway-infra(cc669af9-abc5-4e22-abae-b5e37ef7beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" Nov 03 02:15:14.592061 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:14.592019 4992 scope.go:122] "RemoveContainer" containerID="061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323" Nov 03 02:15:15.249212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:15.249175 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerStarted","Data":"e218327570e9102146299c82d6756866c4a8f129f6dd949beef7eb1f79a8e083"} Nov 03 02:15:15.588326 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:15.588246 4992 scope.go:122] "RemoveContainer" containerID="6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed" Nov 03 02:15:16.258034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.258001 4992 generic.go:356] "Generic (PLEG): container finished" podID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerID="e218327570e9102146299c82d6756866c4a8f129f6dd949beef7eb1f79a8e083" exitCode=60 Nov 03 02:15:16.258531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.258078 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerDied","Data":"e218327570e9102146299c82d6756866c4a8f129f6dd949beef7eb1f79a8e083"} Nov 03 02:15:16.258531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.258166 4992 scope.go:122] "RemoveContainer" containerID="061c4ace8870af0fedcfa7e92ba0adf4e6416954af93752cbbbdff941d7ef323" Nov 03 02:15:16.258657 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.258610 4992 scope.go:122] "RemoveContainer" containerID="e218327570e9102146299c82d6756866c4a8f129f6dd949beef7eb1f79a8e083" Nov 03 02:15:16.258948 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:16.258924 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449_odh-ai-gateway-infra(6840063a-82fe-41ae-b1a7-667ab2e7abea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" Nov 03 02:15:16.260196 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.260173 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerStarted","Data":"461a14d3dd4eea175e4bd40887936d15e6d52cce89015ac6c09321d638143439"} Nov 03 02:15:16.410214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.410174 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:16.588320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.588221 4992 scope.go:122] "RemoveContainer" containerID="42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41" Nov 03 02:15:16.588512 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.588388 4992 scope.go:122] "RemoveContainer" containerID="57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1" Nov 03 02:15:16.588712 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:16.588691 4992 scope.go:122] "RemoveContainer" containerID="5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b" Nov 03 02:15:17.278945 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.278888 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerStarted","Data":"eca91c883a2a115d4de550aca4ab07277d94fea908db17de877d33ace806e411"} Nov 03 02:15:17.280844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.280817 4992 generic.go:356] "Generic (PLEG): container finished" podID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerID="461a14d3dd4eea175e4bd40887936d15e6d52cce89015ac6c09321d638143439" exitCode=60 Nov 03 02:15:17.280976 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.280907 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerDied","Data":"461a14d3dd4eea175e4bd40887936d15e6d52cce89015ac6c09321d638143439"} Nov 03 02:15:17.281030 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.281008 4992 scope.go:122] "RemoveContainer" containerID="6c46bcc749c62af1234a6b7583132df8d7e88e5ad0dcceb864f88028a4b9fbed" Nov 03 02:15:17.281493 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.281468 4992 scope.go:122] "RemoveContainer" containerID="461a14d3dd4eea175e4bd40887936d15e6d52cce89015ac6c09321d638143439" Nov 03 02:15:17.281972 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:17.281945 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6_odh-ai-gateway-infra(613e96cb-0d92-408a-a7c8-19590d1d3761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" Nov 03 02:15:17.283244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.283221 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerStarted","Data":"5dee087c6c0e8c0111a1a186d59eff8b479119a62bb851d3c7762990d6258aa6"} Nov 03 02:15:17.285157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.285137 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerStarted","Data":"82672ae69e6d28e95ffc1f49f271ff8cedd0bd5d6daf6293855aae28a6f5b996"} Nov 03 02:15:17.440705 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.440677 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:17.505753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.505711 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\") pod \"6840063a-82fe-41ae-b1a7-667ab2e7abea\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " Nov 03 02:15:17.505943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.505771 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\" (UniqueName: \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\") pod \"6840063a-82fe-41ae-b1a7-667ab2e7abea\" (UID: \"6840063a-82fe-41ae-b1a7-667ab2e7abea\") " Nov 03 02:15:17.506122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.506098 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca" pod "6840063a-82fe-41ae-b1a7-667ab2e7abea" (UID: "6840063a-82fe-41ae-b1a7-667ab2e7abea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:17.506292 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.506274 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6840063a-82fe-41ae-b1a7-667ab2e7abea-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:17.508097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.508073 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86" pod "6840063a-82fe-41ae-b1a7-667ab2e7abea" (UID: "6840063a-82fe-41ae-b1a7-667ab2e7abea"). InnerVolumeSpecName "kube-api-access-vfw86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:17.581737 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.581706 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:17.587765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.587730 4992 scope.go:122] "RemoveContainer" containerID="dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87" Nov 03 02:15:17.608393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.608349 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfw86\" (UniqueName: \"kubernetes.io/projected/6840063a-82fe-41ae-b1a7-667ab2e7abea-kube-api-access-vfw86\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:17.643284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:17.643218 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" podStartSLOduration=17.643196918 podStartE2EDuration="17.643196918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:17.639243131 +0000 UTC m=+3453.218065719" watchObservedRunningTime="2026-11-03 02:15:17.643196918 +0000 UTC m=+3453.222019506" Nov 03 02:15:18.298207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.298172 4992 generic.go:356] "Generic (PLEG): container finished" podID="f910323a-f6d3-4c46-889c-228b72c58256" containerID="5dee087c6c0e8c0111a1a186d59eff8b479119a62bb851d3c7762990d6258aa6" exitCode=60 Nov 03 02:15:18.298662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.298246 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerDied","Data":"5dee087c6c0e8c0111a1a186d59eff8b479119a62bb851d3c7762990d6258aa6"} Nov 03 02:15:18.298662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.298312 4992 scope.go:122] "RemoveContainer" containerID="5bf2c49f7349bcc0763047afa5eccbd00f091bd909d3f9541bb280aca7a9ff8b" Nov 03 02:15:18.298763 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.298727 4992 scope.go:122] "RemoveContainer" containerID="5dee087c6c0e8c0111a1a186d59eff8b479119a62bb851d3c7762990d6258aa6" Nov 03 02:15:18.299103 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:18.299076 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq_odh-ai-gateway-infra(f910323a-f6d3-4c46-889c-228b72c58256)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" podUID="f910323a-f6d3-4c46-889c-228b72c58256" Nov 03 02:15:18.300680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.300658 4992 generic.go:356] "Generic (PLEG): container finished" podID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerID="82672ae69e6d28e95ffc1f49f271ff8cedd0bd5d6daf6293855aae28a6f5b996" exitCode=60 Nov 03 02:15:18.300833 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.300686 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerDied","Data":"82672ae69e6d28e95ffc1f49f271ff8cedd0bd5d6daf6293855aae28a6f5b996"} Nov 03 02:15:18.301064 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.301047 4992 scope.go:122] "RemoveContainer" containerID="82672ae69e6d28e95ffc1f49f271ff8cedd0bd5d6daf6293855aae28a6f5b996" Nov 03 02:15:18.301399 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:18.301372 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm_odh-ai-gateway-infra(eeb09b4c-0c69-41d7-a942-88f0db86e2b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" Nov 03 02:15:18.302763 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.302742 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerStarted","Data":"092c9d94336a28ab602d9321f11f5d1cb1e9056069b7cf2f0f3e6857c26e7ae3"} Nov 03 02:15:18.304354 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.304320 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" Nov 03 02:15:18.304455 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.304323 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449" event={"ID":"6840063a-82fe-41ae-b1a7-667ab2e7abea","Type":"ContainerDied","Data":"20d5025c78e5a3f37916baabfb2e8456b2d4bcfb6af202653179ed875bda0e1e"} Nov 03 02:15:18.306612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.306588 4992 generic.go:356] "Generic (PLEG): container finished" podID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerID="eca91c883a2a115d4de550aca4ab07277d94fea908db17de877d33ace806e411" exitCode=60 Nov 03 02:15:18.306714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.306640 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerDied","Data":"eca91c883a2a115d4de550aca4ab07277d94fea908db17de877d33ace806e411"} Nov 03 02:15:18.307007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.306988 4992 scope.go:122] "RemoveContainer" containerID="eca91c883a2a115d4de550aca4ab07277d94fea908db17de877d33ace806e411" Nov 03 02:15:18.307280 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:18.307253 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv_odh-ai-gateway-infra(b5932dba-f1a9-49f7-96a4-15ce53a18897)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" Nov 03 02:15:18.315747 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.314439 4992 scope.go:122] "RemoveContainer" containerID="42327a19733ca5be0de82db128f9edd01e0c4d32333ba92852aca52bf006ac41" Nov 03 02:15:18.329329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.329302 4992 scope.go:122] "RemoveContainer" containerID="e218327570e9102146299c82d6756866c4a8f129f6dd949beef7eb1f79a8e083" Nov 03 02:15:18.341335 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.341307 4992 scope.go:122] "RemoveContainer" containerID="57fa0463098683e03357f348cba7d2902684c40a7ee777bdbc2bbbaf7117f5c1" Nov 03 02:15:18.438825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.438798 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:18.520097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.520064 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\") pod \"613e96cb-0d92-408a-a7c8-19590d1d3761\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " Nov 03 02:15:18.520231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.520108 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\" (UniqueName: \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\") pod \"613e96cb-0d92-408a-a7c8-19590d1d3761\" (UID: \"613e96cb-0d92-408a-a7c8-19590d1d3761\") " Nov 03 02:15:18.520522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.520500 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca" pod "613e96cb-0d92-408a-a7c8-19590d1d3761" (UID: "613e96cb-0d92-408a-a7c8-19590d1d3761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:18.562524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.562101 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm" pod "613e96cb-0d92-408a-a7c8-19590d1d3761" (UID: "613e96cb-0d92-408a-a7c8-19590d1d3761"). InnerVolumeSpecName "kube-api-access-5tcbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:18.621467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.621435 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613e96cb-0d92-408a-a7c8-19590d1d3761-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.621467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.621463 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tcbm\" (UniqueName: \"kubernetes.io/projected/613e96cb-0d92-408a-a7c8-19590d1d3761-kube-api-access-5tcbm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:18.652641 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.652606 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:18.769157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.769119 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:18.888216 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:18.888124 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:19.011418 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.011377 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:19.021618 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.021573 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894535-rw449"] Nov 03 02:15:19.316351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.316322 4992 generic.go:356] "Generic (PLEG): container finished" podID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerID="092c9d94336a28ab602d9321f11f5d1cb1e9056069b7cf2f0f3e6857c26e7ae3" exitCode=60 Nov 03 02:15:19.316752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.316395 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerDied","Data":"092c9d94336a28ab602d9321f11f5d1cb1e9056069b7cf2f0f3e6857c26e7ae3"} Nov 03 02:15:19.316752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.316457 4992 scope.go:122] "RemoveContainer" containerID="dd8e6f7255848acf409a03bb9f67f5e2464620ec61589d4b6fe998e956224e87" Nov 03 02:15:19.316909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.316887 4992 scope.go:122] "RemoveContainer" containerID="092c9d94336a28ab602d9321f11f5d1cb1e9056069b7cf2f0f3e6857c26e7ae3" Nov 03 02:15:19.317426 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:19.317396 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894535-ncjsz_odh-ai-gateway-infra(cc669af9-abc5-4e22-abae-b5e37ef7beab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" Nov 03 02:15:19.320765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.320745 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" Nov 03 02:15:19.320857 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.320764 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6" event={"ID":"613e96cb-0d92-408a-a7c8-19590d1d3761","Type":"ContainerDied","Data":"40e6e4cdb9a76e317f27dc5f787142660e1eabb606d057350ae6d8efe05b9993"} Nov 03 02:15:19.334374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.334339 4992 scope.go:122] "RemoveContainer" containerID="461a14d3dd4eea175e4bd40887936d15e6d52cce89015ac6c09321d638143439" Nov 03 02:15:19.524248 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.524214 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:19.554840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.554809 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:19.592479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.592451 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:19.633684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.633650 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\" (UniqueName: \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\") pod \"f910323a-f6d3-4c46-889c-228b72c58256\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " Nov 03 02:15:19.633905 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.633711 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\") pod \"f910323a-f6d3-4c46-889c-228b72c58256\" (UID: \"f910323a-f6d3-4c46-889c-228b72c58256\") " Nov 03 02:15:19.633905 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.633746 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\") pod \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " Nov 03 02:15:19.634025 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.633954 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\" (UniqueName: \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\") pod \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\" (UID: \"eeb09b4c-0c69-41d7-a942-88f0db86e2b4\") " Nov 03 02:15:19.634025 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634012 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\" (UniqueName: \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\") pod \"b5932dba-f1a9-49f7-96a4-15ce53a18897\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " Nov 03 02:15:19.634136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634099 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca" pod "eeb09b4c-0c69-41d7-a942-88f0db86e2b4" (UID: "eeb09b4c-0c69-41d7-a942-88f0db86e2b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:19.634202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca" pod "f910323a-f6d3-4c46-889c-228b72c58256" (UID: "f910323a-f6d3-4c46-889c-228b72c58256"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:19.634202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634170 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\") pod \"b5932dba-f1a9-49f7-96a4-15ce53a18897\" (UID: \"b5932dba-f1a9-49f7-96a4-15ce53a18897\") " Nov 03 02:15:19.634613 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634551 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca" pod "b5932dba-f1a9-49f7-96a4-15ce53a18897" (UID: "b5932dba-f1a9-49f7-96a4-15ce53a18897"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:19.634714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634659 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f910323a-f6d3-4c46-889c-228b72c58256-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.634714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.634681 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.636609 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.636535 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq" pod "f910323a-f6d3-4c46-889c-228b72c58256" (UID: "f910323a-f6d3-4c46-889c-228b72c58256"). InnerVolumeSpecName "kube-api-access-gzmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:19.636609 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.636548 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct" pod "eeb09b4c-0c69-41d7-a942-88f0db86e2b4" (UID: "eeb09b4c-0c69-41d7-a942-88f0db86e2b4"). InnerVolumeSpecName "kube-api-access-ghmct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:19.636845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.636825 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s" pod "b5932dba-f1a9-49f7-96a4-15ce53a18897" (UID: "b5932dba-f1a9-49f7-96a4-15ce53a18897"). InnerVolumeSpecName "kube-api-access-tfg7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:19.671011 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.670966 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:19.677700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.677657 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894535-zv2c6"] Nov 03 02:15:19.736137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.736083 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmxq\" (UniqueName: \"kubernetes.io/projected/f910323a-f6d3-4c46-889c-228b72c58256-kube-api-access-gzmxq\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.736137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.736132 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmct\" (UniqueName: \"kubernetes.io/projected/eeb09b4c-0c69-41d7-a942-88f0db86e2b4-kube-api-access-ghmct\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.736137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.736143 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfg7s\" (UniqueName: \"kubernetes.io/projected/b5932dba-f1a9-49f7-96a4-15ce53a18897-kube-api-access-tfg7s\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.736137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.736152 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5932dba-f1a9-49f7-96a4-15ce53a18897-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:19.786531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:19.786494 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:20.332521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.332488 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" Nov 03 02:15:20.332521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.332507 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm" event={"ID":"eeb09b4c-0c69-41d7-a942-88f0db86e2b4","Type":"ContainerDied","Data":"a2f3f5e10a9c675d937c29c4e4d575ec601401593a529c90551b533a684e64c4"} Nov 03 02:15:20.333155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.332587 4992 scope.go:122] "RemoveContainer" containerID="82672ae69e6d28e95ffc1f49f271ff8cedd0bd5d6daf6293855aae28a6f5b996" Nov 03 02:15:20.337597 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.336419 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" Nov 03 02:15:20.337597 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.336419 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv" event={"ID":"b5932dba-f1a9-49f7-96a4-15ce53a18897","Type":"ContainerDied","Data":"c366733bedb86bd1e56a32a7f8728df3f3158b317dab8d65ba09faffd0cfc3c4"} Nov 03 02:15:20.344386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.344352 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" event={"ID":"f910323a-f6d3-4c46-889c-228b72c58256","Type":"ContainerDied","Data":"c6b2f8471484147001e21c84d25958b6d8f5cee565c227c840d885e8b20cbf27"} Nov 03 02:15:20.344529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.344507 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq" Nov 03 02:15:20.349315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.346441 4992 scope.go:122] "RemoveContainer" containerID="eca91c883a2a115d4de550aca4ab07277d94fea908db17de877d33ace806e411" Nov 03 02:15:20.359716 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.359693 4992 scope.go:122] "RemoveContainer" containerID="5dee087c6c0e8c0111a1a186d59eff8b479119a62bb851d3c7762990d6258aa6" Nov 03 02:15:20.477373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477345 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w"] Nov 03 02:15:20.477908 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477891 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.477950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477909 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.477950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477921 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.477950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477926 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.477950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477938 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.477950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477944 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477955 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477959 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477971 4992 cpu_manager.go:414] "containerMap: removing container" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477976 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477993 4992 cpu_manager.go:414] "containerMap: removing container" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.477997 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478003 4992 cpu_manager.go:414] "containerMap: removing container" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478007 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478030 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478034 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478040 4992 cpu_manager.go:414] "containerMap: removing container" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478044 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478056 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478062 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478079 4992 cpu_manager.go:414] "containerMap: removing container" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478084 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478173 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478183 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478189 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478205 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478211 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478218 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478225 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478232 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478241 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:20.478635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.478248 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:20.483316 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.483292 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.485117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.485096 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:20.506899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.506859 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:20.510018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.509987 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894535-dqlmm"] Nov 03 02:15:20.534871 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.533939 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w"] Nov 03 02:15:20.537922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.537856 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:15:20.544447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.544414 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\" (UniqueName: \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\") pod \"cc669af9-abc5-4e22-abae-b5e37ef7beab\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " Nov 03 02:15:20.544717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.544693 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\") pod \"cc669af9-abc5-4e22-abae-b5e37ef7beab\" (UID: \"cc669af9-abc5-4e22-abae-b5e37ef7beab\") " Nov 03 02:15:20.545061 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.545034 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5rg\" (UniqueName: \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.545151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.545124 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca" pod "cc669af9-abc5-4e22-abae-b5e37ef7beab" (UID: "cc669af9-abc5-4e22-abae-b5e37ef7beab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:20.545249 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.545229 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.545315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.545303 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.545488 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.545467 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc669af9-abc5-4e22-abae-b5e37ef7beab-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.546700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.546676 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m" pod "cc669af9-abc5-4e22-abae-b5e37ef7beab" (UID: "cc669af9-abc5-4e22-abae-b5e37ef7beab"). InnerVolumeSpecName "kube-api-access-q9h6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:20.596032 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.595948 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" path="/var/lib/kubelet/pods/613e96cb-0d92-408a-a7c8-19590d1d3761/volumes" Nov 03 02:15:20.596451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.596434 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" path="/var/lib/kubelet/pods/6840063a-82fe-41ae-b1a7-667ab2e7abea/volumes" Nov 03 02:15:20.596882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.596867 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" path="/var/lib/kubelet/pods/eeb09b4c-0c69-41d7-a942-88f0db86e2b4/volumes" Nov 03 02:15:20.646410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.646375 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.646633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.646429 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.646633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.646592 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5rg\" (UniqueName: \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.646727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.646689 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9h6m\" (UniqueName: \"kubernetes.io/projected/cc669af9-abc5-4e22-abae-b5e37ef7beab-kube-api-access-q9h6m\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:20.647285 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.647257 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.649269 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.649241 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.710949 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.710907 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5rg\" (UniqueName: \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\") pod \"maas-api-revoke-sub-93626c15-vmr6w\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.741217 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.741180 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:20.744648 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.744619 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894535-g4zqq"] Nov 03 02:15:20.796161 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.796122 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:20.858830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.858794 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:20.862855 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:20.862826 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894535-nb2rv"] Nov 03 02:15:21.176486 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:15:21.176451 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97139aa_4476_47ae_ab97_98ba71f415db.slice/crio-0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257 WatchSource:0}: Error finding container 0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257: Status 404 returned error can't find the container with id 0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257 Nov 03 02:15:21.179480 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.179431 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w"] Nov 03 02:15:21.352224 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.352187 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" Nov 03 02:15:21.352739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.352219 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz" event={"ID":"cc669af9-abc5-4e22-abae-b5e37ef7beab","Type":"ContainerDied","Data":"6744ae8aa81ce9ad9596ab6f5e834e86c4b1b50ae20ccb8764a305aa576b75dd"} Nov 03 02:15:21.352739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.352297 4992 scope.go:122] "RemoveContainer" containerID="092c9d94336a28ab602d9321f11f5d1cb1e9056069b7cf2f0f3e6857c26e7ae3" Nov 03 02:15:21.357860 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.357804 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" event={"ID":"e97139aa-4476-47ae-ab97-98ba71f415db","Type":"ContainerStarted","Data":"0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257"} Nov 03 02:15:21.526811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.526772 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:21.530437 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:21.530404 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894535-ncjsz"] Nov 03 02:15:22.366200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:22.366114 4992 generic.go:356] "Generic (PLEG): container finished" podID="e97139aa-4476-47ae-ab97-98ba71f415db" containerID="d2afa483cf2606c7a398a4783c826fdda1f4a331f1dc288f2907448cfc0b579d" exitCode=0 Nov 03 02:15:22.366684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:22.366199 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" event={"ID":"e97139aa-4476-47ae-ab97-98ba71f415db","Type":"ContainerDied","Data":"d2afa483cf2606c7a398a4783c826fdda1f4a331f1dc288f2907448cfc0b579d"} Nov 03 02:15:22.595806 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:22.595771 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" path="/var/lib/kubelet/pods/b5932dba-f1a9-49f7-96a4-15ce53a18897/volumes" Nov 03 02:15:22.596221 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:22.596208 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" path="/var/lib/kubelet/pods/cc669af9-abc5-4e22-abae-b5e37ef7beab/volumes" Nov 03 02:15:22.596626 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:22.596610 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f910323a-f6d3-4c46-889c-228b72c58256" path="/var/lib/kubelet/pods/f910323a-f6d3-4c46-889c-228b72c58256/volumes" Nov 03 02:15:23.543514 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.543477 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:23.580249 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.580199 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\") pod \"e97139aa-4476-47ae-ab97-98ba71f415db\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " Nov 03 02:15:23.580422 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.580338 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\") pod \"e97139aa-4476-47ae-ab97-98ba71f415db\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " Nov 03 02:15:23.580468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.580442 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\" (UniqueName: \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\") pod \"e97139aa-4476-47ae-ab97-98ba71f415db\" (UID: \"e97139aa-4476-47ae-ab97-98ba71f415db\") " Nov 03 02:15:23.580799 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.580770 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca" pod "e97139aa-4476-47ae-ab97-98ba71f415db" (UID: "e97139aa-4476-47ae-ab97-98ba71f415db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:23.580998 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.580978 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:23.582333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.582313 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca" pod "e97139aa-4476-47ae-ab97-98ba71f415db" (UID: "e97139aa-4476-47ae-ab97-98ba71f415db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:15:23.582509 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.582489 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg" pod "e97139aa-4476-47ae-ab97-98ba71f415db" (UID: "e97139aa-4476-47ae-ab97-98ba71f415db"). InnerVolumeSpecName "kube-api-access-fk5rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:23.682860 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.682787 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97139aa-4476-47ae-ab97-98ba71f415db-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:23.682860 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:23.682822 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5rg\" (UniqueName: \"kubernetes.io/projected/e97139aa-4476-47ae-ab97-98ba71f415db-kube-api-access-fk5rg\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:24.382650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:24.382573 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" Nov 03 02:15:24.382650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:24.382590 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w" event={"ID":"e97139aa-4476-47ae-ab97-98ba71f415db","Type":"ContainerDied","Data":"0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257"} Nov 03 02:15:24.382650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:24.382646 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2dc9b924125372856aa1313b38a7fc440c614e0b5d50e2598e3490163a4257" Nov 03 02:15:31.233938 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.233886 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg"] Nov 03 02:15:31.234490 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234472 4992 cpu_manager.go:414] "containerMap: removing container" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234492 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234501 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234507 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234527 4992 cpu_manager.go:414] "containerMap: removing container" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234533 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234548 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234552 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:31.234552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234583 4992 cpu_manager.go:414] "containerMap: removing container" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234588 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234596 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e97139aa-4476-47ae-ab97-98ba71f415db" containerName="revoke-keys" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234600 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97139aa-4476-47ae-ab97-98ba71f415db" containerName="revoke-keys" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234606 4992 cpu_manager.go:414] "containerMap: removing container" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234611 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234703 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="613e96cb-0d92-408a-a7c8-19590d1d3761" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234715 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234723 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234729 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="f910323a-f6d3-4c46-889c-228b72c58256" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234737 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6840063a-82fe-41ae-b1a7-667ab2e7abea" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234742 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5932dba-f1a9-49f7-96a4-15ce53a18897" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234749 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97139aa-4476-47ae-ab97-98ba71f415db" containerName="revoke-keys" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234755 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:31.234830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.234761 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb09b4c-0c69-41d7-a942-88f0db86e2b4" containerName="cleanup" Nov 03 02:15:31.238044 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.238025 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.291966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.291927 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg"] Nov 03 02:15:31.339115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.339080 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:15:31.339115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.339088 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:15:31.376653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.376612 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mpx\" (UniqueName: \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.376861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.376664 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.376861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.376800 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.478283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.478233 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mpx\" (UniqueName: \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.478495 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.478298 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.478495 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.478342 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.479130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.479108 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.480847 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.480825 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.541248 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.541159 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mpx\" (UniqueName: \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\") pod \"maas-api-revoke-sub-caccfcd2-wqfrg\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.551370 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.551337 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:31.936413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:31.936383 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg"] Nov 03 02:15:32.152427 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:32.152381 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:15:32.435697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:32.435653 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" event={"ID":"e4fa0cc9-37e1-4589-b4be-48f718e028fa","Type":"ContainerStarted","Data":"b6b8060abe0a07f790c7b5054ae4009fce42b79d730e73fa98988f40593ed097"} Nov 03 02:15:32.436118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:32.435750 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" event={"ID":"e4fa0cc9-37e1-4589-b4be-48f718e028fa","Type":"ContainerStarted","Data":"09a70c325175eace9ff4e03b729045d44091e1f264592879de255b0afb85b88b"} Nov 03 02:15:32.555118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:32.554996 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" podStartSLOduration=1.5549825400000001 podStartE2EDuration="1.55498254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:32.554329737 +0000 UTC m=+3468.133152323" watchObservedRunningTime="2026-11-03 02:15:32.55498254 +0000 UTC m=+3468.133805126" Nov 03 02:15:33.443694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:33.443653 4992 generic.go:356] "Generic (PLEG): container finished" podID="e4fa0cc9-37e1-4589-b4be-48f718e028fa" containerID="b6b8060abe0a07f790c7b5054ae4009fce42b79d730e73fa98988f40593ed097" exitCode=0 Nov 03 02:15:33.444078 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:33.443706 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" event={"ID":"e4fa0cc9-37e1-4589-b4be-48f718e028fa","Type":"ContainerDied","Data":"b6b8060abe0a07f790c7b5054ae4009fce42b79d730e73fa98988f40593ed097"} Nov 03 02:15:34.617065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.617033 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:34.717037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.716948 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\") pod \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " Nov 03 02:15:34.717235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.717122 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\") pod \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " Nov 03 02:15:34.717322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.717301 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\" (UniqueName: \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\") pod \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\" (UID: \"e4fa0cc9-37e1-4589-b4be-48f718e028fa\") " Nov 03 02:15:34.717382 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.717336 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca" pod "e4fa0cc9-37e1-4589-b4be-48f718e028fa" (UID: "e4fa0cc9-37e1-4589-b4be-48f718e028fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:15:34.717828 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.717800 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:34.720045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.720014 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx" pod "e4fa0cc9-37e1-4589-b4be-48f718e028fa" (UID: "e4fa0cc9-37e1-4589-b4be-48f718e028fa"). InnerVolumeSpecName "kube-api-access-g4mpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:15:34.720207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.720191 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca" pod "e4fa0cc9-37e1-4589-b4be-48f718e028fa" (UID: "e4fa0cc9-37e1-4589-b4be-48f718e028fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:15:34.818849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.818806 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mpx\" (UniqueName: \"kubernetes.io/projected/e4fa0cc9-37e1-4589-b4be-48f718e028fa-kube-api-access-g4mpx\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:34.818849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:34.818837 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fa0cc9-37e1-4589-b4be-48f718e028fa-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:15:35.459376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:35.459342 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" Nov 03 02:15:35.459376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:35.459358 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg" event={"ID":"e4fa0cc9-37e1-4589-b4be-48f718e028fa","Type":"ContainerDied","Data":"09a70c325175eace9ff4e03b729045d44091e1f264592879de255b0afb85b88b"} Nov 03 02:15:35.459674 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:35.459407 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a70c325175eace9ff4e03b729045d44091e1f264592879de255b0afb85b88b" Nov 03 02:15:56.936605 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:56.936552 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:15:58.895653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:58.895604 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:15:59.304761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.304722 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:15:59.305244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.305230 4992 cpu_manager.go:414] "containerMap: removing container" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:59.305286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.305245 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc669af9-abc5-4e22-abae-b5e37ef7beab" containerName="cleanup" Nov 03 02:15:59.305286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.305274 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e4fa0cc9-37e1-4589-b4be-48f718e028fa" containerName="revoke-keys" Nov 03 02:15:59.305286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.305280 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fa0cc9-37e1-4589-b4be-48f718e028fa" containerName="revoke-keys" Nov 03 02:15:59.305379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.305375 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fa0cc9-37e1-4589-b4be-48f718e028fa" containerName="revoke-keys" Nov 03 02:15:59.309122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.309100 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.365788 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.365755 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c70365-serving-cert\"" Nov 03 02:15:59.365982 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.365938 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:15:59.440759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.440707 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhslk\" (UniqueName: \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.440953 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.440850 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.441034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.441010 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.441094 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.441062 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.542909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.542869 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.542909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.542920 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.543210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.543039 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhslk\" (UniqueName: \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.543210 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:59.543069 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-serving-cert: secret "maas-api-e2e-extmod-a-c70365-serving-cert" not found Nov 03 02:15:59.543210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.543088 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.543210 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:15:59.543203 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls podName:b357d84b-022b-42d3-b485-4407078db4d9 nodeName:}" failed. No retries permitted until 2026-11-03 02:16:00.04317923 +0000 UTC m=+3495.622001810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls") pod "maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" (UID: "b357d84b-022b-42d3-b485-4407078db4d9") : secret "maas-api-e2e-extmod-a-c70365-serving-cert" not found Nov 03 02:15:59.544033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.544002 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.547242 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.547215 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:15:59.606696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:15:59.606610 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhslk\" (UniqueName: \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:00.049651 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:00.049615 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:00.052396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:00.052369 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:00.222450 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:00.222395 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:00.666516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:00.666484 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:16:00.674384 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:00.672595 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:16:01.686059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:01.686021 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" event={"ID":"b357d84b-022b-42d3-b485-4407078db4d9","Type":"ContainerStarted","Data":"c4b8e30d2e7c43e637b87bed5a9a49f0440353dcda832ab1dbbe31bb03c1b757"} Nov 03 02:16:02.152361 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:16:02.152313 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:16:02.427510 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.427462 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:16:02.431680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.431651 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.487103 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.487069 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-aab75e-openshift-default-dockercfg-7xknm\"" Nov 03 02:16:02.544086 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.544040 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592427 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592517 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592581 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592623 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592661 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592736 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592877 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw7w\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592911 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.599579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.592943 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.693919 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.693831 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694037 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw7w\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694072 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694101 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694187 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694265 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694306 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694355 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694424 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.694921 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.694897 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.695046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.695028 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.695981 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.695953 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.696224 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.696198 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.696453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.696430 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.697431 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.697404 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" event={"ID":"b357d84b-022b-42d3-b485-4407078db4d9","Type":"ContainerStarted","Data":"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16"} Nov 03 02:16:02.697923 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.697890 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:02.698239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.698216 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.698717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.698696 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.757364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.757326 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.758244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.758208 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw7w\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\") pod \"e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:02.820484 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:02.820415 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" podStartSLOduration=2.841486789 podStartE2EDuration="3.820398588s" totalImagesPullingTime="978.911799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:02.819120679 +0000 UTC m=+3498.397943266" watchObservedRunningTime="2026-11-03 02:16:02.820398588 +0000 UTC m=+3498.399221203" Nov 03 02:16:03.047195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.047105 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:03.587889 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:16:03.587858 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d5d275_c7f7_499a_9f63_764451a91487.slice/crio-12c2472d26b9b3ff97ec9922a7195073630a8d653226641d5bf98747497274da WatchSource:0}: Error finding container 12c2472d26b9b3ff97ec9922a7195073630a8d653226641d5bf98747497274da: Status 404 returned error can't find the container with id 12c2472d26b9b3ff97ec9922a7195073630a8d653226641d5bf98747497274da Nov 03 02:16:03.590758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.590725 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:03.590851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.590793 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:03.590851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.590823 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:03.591040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.591019 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:16:03.710878 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.710824 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerStarted","Data":"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a"} Nov 03 02:16:03.711617 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.710886 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerStarted","Data":"12c2472d26b9b3ff97ec9922a7195073630a8d653226641d5bf98747497274da"} Nov 03 02:16:03.836686 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:03.836572 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podStartSLOduration=1.836541559 podStartE2EDuration="1.836541559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:03.83560614 +0000 UTC m=+3499.414428727" watchObservedRunningTime="2026-11-03 02:16:03.836541559 +0000 UTC m=+3499.415364198" Nov 03 02:16:04.048287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:04.048250 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:04.050281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:04.050248 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:16:04.050420 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:04.050313 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:16:05.052646 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:05.052608 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:05.723655 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:05.723620 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:05.724648 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:05.724628 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:16:09.724293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:09.724257 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:16:11.957591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:11.957518 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" podUID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" containerName="main" containerID="cri-o://4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b" gracePeriod=45 Nov 03 02:16:12.233188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.233160 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:16:12.311240 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311190 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\" (UniqueName: \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311439 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311293 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311439 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311331 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311439 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311363 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311519 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311622 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\") pod \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\" (UID: \"d311a889-6fe4-4266-bc01-198ac2a8e3fc\") " Nov 03 02:16:12.311767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311682 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:12.311767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311742 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:12.311879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.311827 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:12.312131 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.312101 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.312244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.312133 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.312244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.312146 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.313802 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.313770 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:16:12.313891 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.313781 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:12.313891 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.313813 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr" pod "d311a889-6fe4-4266-bc01-198ac2a8e3fc" (UID: "d311a889-6fe4-4266-bc01-198ac2a8e3fc"). InnerVolumeSpecName "kube-api-access-8rgbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:12.413289 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.413243 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a889-6fe4-4266-bc01-198ac2a8e3fc-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.413289 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.413281 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rgbr\" (UniqueName: \"kubernetes.io/projected/d311a889-6fe4-4266-bc01-198ac2a8e3fc-kube-api-access-8rgbr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.413289 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.413292 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a889-6fe4-4266-bc01-198ac2a8e3fc-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:12.771811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.771773 4992 generic.go:356] "Generic (PLEG): container finished" podID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" containerID="4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b" exitCode=0 Nov 03 02:16:12.771987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.771815 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" Nov 03 02:16:12.771987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.771858 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" event={"ID":"d311a889-6fe4-4266-bc01-198ac2a8e3fc","Type":"ContainerDied","Data":"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b"} Nov 03 02:16:12.771987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.771897 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r" event={"ID":"d311a889-6fe4-4266-bc01-198ac2a8e3fc","Type":"ContainerDied","Data":"f74e86613a2e0479a2524d263a1ec88daf5a371bf78c1e1ed02b3cc7f763e871"} Nov 03 02:16:12.771987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.771916 4992 scope.go:122] "RemoveContainer" containerID="4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b" Nov 03 02:16:12.781829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.781807 4992 scope.go:122] "RemoveContainer" containerID="4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b" Nov 03 02:16:12.782172 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:16:12.782147 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b\": container with ID starting with 4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b not found: ID does not exist" containerID="4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b" Nov 03 02:16:12.782230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.782186 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b"} err="failed to get container status \"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b\": rpc error: code = NotFound desc = could not find container \"4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b\": container with ID starting with 4665f7033e002366ec7aa2071d92849aa51bf8c2af7701685951ca84c96bcc3b not found: ID does not exist" Nov 03 02:16:12.946579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.946526 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:16:12.951770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:12.951735 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1c9b4e/auth-test-model-1c9b4e-kserve-8b746b69c-bt64r"] Nov 03 02:16:13.923377 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:13.923330 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerName="main" containerID="cri-o://53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850" gracePeriod=45 Nov 03 02:16:14.025891 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.025844 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.127:8000/ready\": dial tcp 10.132.0.127:8000: connect: connection refused" Nov 03 02:16:14.228440 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.228412 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:16:14.354845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.354682 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.354845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.354736 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\" (UniqueName: \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.355241 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.355200 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\" (UniqueName: \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.356500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.355406 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.356500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.355546 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.356500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.355593 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\") pod \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\" (UID: \"50f86c8c-d38b-4741-83f2-f1cd5821f70a\") " Nov 03 02:16:14.357654 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.356925 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:14.358196 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.357861 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:14.378615 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.378184 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:14.378615 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.378350 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "kube-api-access-rsk2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:16:14.400654 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.385985 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:16:14.407941 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.407890 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm" pod "50f86c8c-d38b-4741-83f2-f1cd5821f70a" (UID: "50f86c8c-d38b-4741-83f2-f1cd5821f70a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457874 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457908 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457919 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457933 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f86c8c-d38b-4741-83f2-f1cd5821f70a-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457946 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f86c8c-d38b-4741-83f2-f1cd5821f70a-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.457964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.457960 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsk2d\" (UniqueName: \"kubernetes.io/projected/50f86c8c-d38b-4741-83f2-f1cd5821f70a-kube-api-access-rsk2d\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:16:14.604288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.604251 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" path="/var/lib/kubelet/pods/d311a889-6fe4-4266-bc01-198ac2a8e3fc/volumes" Nov 03 02:16:14.789694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.789587 4992 generic.go:356] "Generic (PLEG): container finished" podID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerID="53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850" exitCode=0 Nov 03 02:16:14.789865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.789708 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" Nov 03 02:16:14.789865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.789725 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" event={"ID":"50f86c8c-d38b-4741-83f2-f1cd5821f70a","Type":"ContainerDied","Data":"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850"} Nov 03 02:16:14.789865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.789768 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf" event={"ID":"50f86c8c-d38b-4741-83f2-f1cd5821f70a","Type":"ContainerDied","Data":"ae5c98ca69e8515678c2200726154b967055145c68c007b8aefbba55f5c5aa7f"} Nov 03 02:16:14.789865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.789785 4992 scope.go:122] "RemoveContainer" containerID="53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850" Nov 03 02:16:14.799672 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.799649 4992 scope.go:122] "RemoveContainer" containerID="53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850" Nov 03 02:16:14.800001 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:16:14.799980 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850\": container with ID starting with 53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850 not found: ID does not exist" containerID="53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850" Nov 03 02:16:14.800055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.800013 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850"} err="failed to get container status \"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850\": rpc error: code = NotFound desc = could not find container \"53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850\": container with ID starting with 53421714a1564139fec333838b015cc88818653e37b40e6350ae50b32d5ce850 not found: ID does not exist" Nov 03 02:16:14.960875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.960834 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:16:14.966446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:14.966413 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-c34879/auth-test-model-c34879-kserve-749cf9d44f-hqqhf"] Nov 03 02:16:16.595333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:16.595295 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" path="/var/lib/kubelet/pods/50f86c8c-d38b-4741-83f2-f1cd5821f70a/volumes" Nov 03 02:16:31.553105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.553061 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:16:31.553907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.553884 4992 cpu_manager.go:414] "containerMap: removing container" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerName="main" Nov 03 02:16:31.553907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.553907 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerName="main" Nov 03 02:16:31.554116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.553967 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" containerName="main" Nov 03 02:16:31.554116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.553977 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" containerName="main" Nov 03 02:16:31.554219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.554143 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f86c8c-d38b-4741-83f2-f1cd5821f70a" containerName="main" Nov 03 02:16:31.554219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.554168 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d311a889-6fe4-4266-bc01-198ac2a8e3fc" containerName="main" Nov 03 02:16:31.558016 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.557992 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.657580 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.655415 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d8cc55-openshift-default-dockercfg-48rtp\"" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668375 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668458 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkvf\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668490 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668571 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668622 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668657 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668775 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668817 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.669619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.668875 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.736820 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.736769 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:16:31.770296 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770257 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770307 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770347 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770424 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770499 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkvf\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770529 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770520 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770603 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770649 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770672 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.770995 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.770828 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.771148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.771124 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.771284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.771261 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.771351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.771281 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.771714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.771691 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.773301 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.773275 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.773591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.773552 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.834734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.834655 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.842402 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.842345 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkvf\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\") pod \"e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:31.872918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:31.872873 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:32.153846 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:16:32.153767 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:16:32.314625 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:16:32.314554 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663cf534_2777_4e07_ae36_bf7c7464dbb2.slice/crio-90ac22529f1b8b5777161e1f7cf7f73fe44a69148c315354e814aa24147e51b8 WatchSource:0}: Error finding container 90ac22529f1b8b5777161e1f7cf7f73fe44a69148c315354e814aa24147e51b8: Status 404 returned error can't find the container with id 90ac22529f1b8b5777161e1f7cf7f73fe44a69148c315354e814aa24147e51b8 Nov 03 02:16:32.317468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.317441 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:16:32.317801 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.317769 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:32.317879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.317835 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:32.317879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.317866 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:32.908855 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.908819 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerStarted","Data":"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283"} Nov 03 02:16:32.909386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:32.908882 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerStarted","Data":"90ac22529f1b8b5777161e1f7cf7f73fe44a69148c315354e814aa24147e51b8"} Nov 03 02:16:33.033971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:33.033908 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podStartSLOduration=2.033891784 podStartE2EDuration="2.033891784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:33.032811645 +0000 UTC m=+3528.611634234" watchObservedRunningTime="2026-11-03 02:16:33.033891784 +0000 UTC m=+3528.612714379" Nov 03 02:16:33.873682 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:33.873637 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:33.881502 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:33.881462 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:33.915851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:33.915819 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:33.916773 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:33.916750 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:16:36.280927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.280884 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:16:36.286623 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.286541 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322233 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322186 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322248 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322337 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322380 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322497 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322547 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322693 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrpsz\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322741 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322723 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.322775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.322747 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.342839 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.342801 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b16721-openshift-default-dockercfg-w8nqd\"" Nov 03 02:16:36.397348 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.397298 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:16:36.423524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423484 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrpsz\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423531 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423596 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423635 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423661 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423683 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423713 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423868 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423960 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.423999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.423983 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.424103 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.424064 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.424221 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.424197 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.424335 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.424244 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.424817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.424791 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.426326 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.426305 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.426400 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.426363 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.489272 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.489231 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.489554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.489528 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrpsz\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-p6f9r\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:36.601638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:36.601532 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:37.032684 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:16:37.032645 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8ad73e_49b9_485c_8906_2bacda1c9567.slice/crio-3aa58bedb36a301c688fcd132891af00bba9db2feee3a65d7cad1f74819baf7d WatchSource:0}: Error finding container 3aa58bedb36a301c688fcd132891af00bba9db2feee3a65d7cad1f74819baf7d: Status 404 returned error can't find the container with id 3aa58bedb36a301c688fcd132891af00bba9db2feee3a65d7cad1f74819baf7d Nov 03 02:16:37.034408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.034381 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:16:37.035978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.035943 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:37.036071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.036041 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:37.036110 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.036071 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:16:37.942329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.942289 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" event={"ID":"ad8ad73e-49b9-485c-8906-2bacda1c9567","Type":"ContainerStarted","Data":"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247"} Nov 03 02:16:37.942849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:37.942345 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" event={"ID":"ad8ad73e-49b9-485c-8906-2bacda1c9567","Type":"ContainerStarted","Data":"3aa58bedb36a301c688fcd132891af00bba9db2feee3a65d7cad1f74819baf7d"} Nov 03 02:16:38.087936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:38.087853 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podStartSLOduration=2.087828177 podStartE2EDuration="2.087828177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:38.083078517 +0000 UTC m=+3533.661901107" watchObservedRunningTime="2026-11-03 02:16:38.087828177 +0000 UTC m=+3533.666650765" Nov 03 02:16:38.602466 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:38.602429 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:38.604349 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:38.604319 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:38.604572 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:38.604383 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:39.602918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:39.602854 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:39.603351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:39.602942 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:40.601920 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:40.601878 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:40.602129 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:40.601936 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:41.602309 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:41.602271 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:41.602777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:41.602342 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:42.602485 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:42.602451 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:42.602918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:42.602508 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:43.602166 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:43.602123 4992 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 02:16:43.602359 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:43.602183 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 02:16:44.607094 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:44.607057 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:44.607499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:44.607349 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:44.608013 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:44.607994 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:16:59.233188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:16:59.233081 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:17:02.151402 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:17:02.151362 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:17:02.243105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:02.243065 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:17:02.243538 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:02.243358 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" containerID="cri-o://d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247" gracePeriod=30 Nov 03 02:17:07.516761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.516690 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:17:07.654778 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654743 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.654980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654786 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.654980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654849 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.654980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654907 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.654980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654927 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.654980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.654975 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.655274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655044 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.655274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655050 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:07.655274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655108 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.655274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655151 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\") pod \"ad8ad73e-49b9-485c-8906-2bacda1c9567\" (UID: \"ad8ad73e-49b9-485c-8906-2bacda1c9567\") " Nov 03 02:17:07.655274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655182 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:07.655548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655410 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:17:07.655874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655676 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.655874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655701 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ad73e-49b9-485c-8906-2bacda1c9567-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.655874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655716 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.655874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655677 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:07.655874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.655831 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:07.657708 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.657685 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:17:07.657942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.657922 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "kube-api-access-mrpsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:07.658141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.658123 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:07.658951 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.658928 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy" pod "ad8ad73e-49b9-485c-8906-2bacda1c9567" (UID: "ad8ad73e-49b9-485c-8906-2bacda1c9567"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:07.757345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757307 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.757345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757339 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.757345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757348 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.757345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757358 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrpsz\" (UniqueName: \"kubernetes.io/projected/ad8ad73e-49b9-485c-8906-2bacda1c9567-kube-api-access-mrpsz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.757345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757368 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:07.757786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:07.757376 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8ad73e-49b9-485c-8906-2bacda1c9567-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:08.186623 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.186552 4992 generic.go:356] "Generic (PLEG): container finished" podID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerID="d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247" exitCode=0 Nov 03 02:17:08.186808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.186644 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" Nov 03 02:17:08.186808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.186672 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" event={"ID":"ad8ad73e-49b9-485c-8906-2bacda1c9567","Type":"ContainerDied","Data":"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247"} Nov 03 02:17:08.186808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.186724 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r" event={"ID":"ad8ad73e-49b9-485c-8906-2bacda1c9567","Type":"ContainerDied","Data":"3aa58bedb36a301c688fcd132891af00bba9db2feee3a65d7cad1f74819baf7d"} Nov 03 02:17:08.186808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.186752 4992 scope.go:122] "RemoveContainer" containerID="d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247" Nov 03 02:17:08.198791 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.198767 4992 scope.go:122] "RemoveContainer" containerID="d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247" Nov 03 02:17:08.199100 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:17:08.199079 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247\": container with ID starting with d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247 not found: ID does not exist" containerID="d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247" Nov 03 02:17:08.199186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.199109 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247"} err="failed to get container status \"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247\": rpc error: code = NotFound desc = could not find container \"d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247\": container with ID starting with d7e6a4561e84dd11a55defa2c2b8f6196ab5d49c5edb64c1d32eaecfc575d247 not found: ID does not exist" Nov 03 02:17:08.360605 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.360551 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:17:08.412868 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.412826 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-p6f9r"] Nov 03 02:17:08.595492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:08.595413 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" path="/var/lib/kubelet/pods/ad8ad73e-49b9-485c-8906-2bacda1c9567/volumes" Nov 03 02:17:14.257208 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.257136 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" podUID="98ab19aa-1701-4407-83b7-59ac03f1b64a" containerName="main" containerID="cri-o://c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b" gracePeriod=45 Nov 03 02:17:14.536430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.536399 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:17:14.640549 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640510 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.640766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640694 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.640766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640739 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\" (UniqueName: \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.640887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640766 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.640887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640833 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.640887 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640840 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:14.641040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640906 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\" (UniqueName: \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\") pod \"98ab19aa-1701-4407-83b7-59ac03f1b64a\" (UID: \"98ab19aa-1701-4407-83b7-59ac03f1b64a\") " Nov 03 02:17:14.641040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.640956 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:14.641167 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.641136 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:14.641409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.641391 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:14.641483 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.641442 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:14.641483 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.641459 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:14.643144 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.643118 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:17:14.643260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.643166 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:17:14.643260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.643233 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w" pod "98ab19aa-1701-4407-83b7-59ac03f1b64a" (UID: "98ab19aa-1701-4407-83b7-59ac03f1b64a"). InnerVolumeSpecName "kube-api-access-klk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:17:14.742413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.742372 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klk4w\" (UniqueName: \"kubernetes.io/projected/98ab19aa-1701-4407-83b7-59ac03f1b64a-kube-api-access-klk4w\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:14.742413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.742408 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ab19aa-1701-4407-83b7-59ac03f1b64a-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:14.742413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:14.742420 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ab19aa-1701-4407-83b7-59ac03f1b64a-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:17:15.237782 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.237747 4992 generic.go:356] "Generic (PLEG): container finished" podID="98ab19aa-1701-4407-83b7-59ac03f1b64a" containerID="c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b" exitCode=0 Nov 03 02:17:15.237994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.237795 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" Nov 03 02:17:15.237994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.237832 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" event={"ID":"98ab19aa-1701-4407-83b7-59ac03f1b64a","Type":"ContainerDied","Data":"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b"} Nov 03 02:17:15.237994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.237886 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz" event={"ID":"98ab19aa-1701-4407-83b7-59ac03f1b64a","Type":"ContainerDied","Data":"a6ae1d6cfecbecedc22dbe7c65999779ff76d73b9fc92da1a8a839f5d950fa30"} Nov 03 02:17:15.237994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.237911 4992 scope.go:122] "RemoveContainer" containerID="c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b" Nov 03 02:17:15.248318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.248297 4992 scope.go:122] "RemoveContainer" containerID="c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b" Nov 03 02:17:15.248669 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:17:15.248644 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b\": container with ID starting with c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b not found: ID does not exist" containerID="c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b" Nov 03 02:17:15.248727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.248685 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b"} err="failed to get container status \"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b\": rpc error: code = NotFound desc = could not find container \"c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b\": container with ID starting with c2647220363d0dbf6f32b5814950a7d6bb506ef44b88d714984b6abaa5c8d13b not found: ID does not exist" Nov 03 02:17:15.409206 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.409162 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:17:15.413122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:15.413093 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9576a/auth-test-model-d9576a-kserve-655c5588f-ksclz"] Nov 03 02:17:16.594703 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:16.594665 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ab19aa-1701-4407-83b7-59ac03f1b64a" path="/var/lib/kubelet/pods/98ab19aa-1701-4407-83b7-59ac03f1b64a/volumes" Nov 03 02:17:32.152713 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:17:32.152671 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:17:44.603413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.603369 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:17:44.603994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.603975 4992 cpu_manager.go:414] "containerMap: removing container" podUID="98ab19aa-1701-4407-83b7-59ac03f1b64a" containerName="main" Nov 03 02:17:44.604041 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.603996 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ab19aa-1701-4407-83b7-59ac03f1b64a" containerName="main" Nov 03 02:17:44.604077 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.604042 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" Nov 03 02:17:44.604077 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.604050 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" Nov 03 02:17:44.604169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.604158 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8ad73e-49b9-485c-8906-2bacda1c9567" containerName="istio-proxy" Nov 03 02:17:44.604209 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.604172 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ab19aa-1701-4407-83b7-59ac03f1b64a" containerName="main" Nov 03 02:17:44.607766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.607746 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.662673 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.662639 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:17:44.710382 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.710352 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-50ca6d-serving-cert\"" Nov 03 02:17:44.723068 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.723033 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.723226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.723096 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.723333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.723312 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.723542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.723516 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvg5\" (UniqueName: \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.824430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.824395 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvg5\" (UniqueName: \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.824696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.824515 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.824696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.824554 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.824696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.824638 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.825218 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.825185 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.827121 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.827094 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.827166 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.887604 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.887510 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvg5\" (UniqueName: \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\") pod \"maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:44.921891 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:44.921851 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:45.117988 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:45.117957 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:17:45.120242 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:45.120216 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:17:45.300369 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:17:45.300329 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2661e6c5_9fff_40fd_9f47_bcc7b6ba0a96.slice/crio-7d7f020e082032ee1889b99f405894c3b1b23b7cab5c8f52e84bef7fb7ff3f91 WatchSource:0}: Error finding container 7d7f020e082032ee1889b99f405894c3b1b23b7cab5c8f52e84bef7fb7ff3f91: Status 404 returned error can't find the container with id 7d7f020e082032ee1889b99f405894c3b1b23b7cab5c8f52e84bef7fb7ff3f91 Nov 03 02:17:45.303151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:45.303124 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:17:45.434139 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:45.434095 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" event={"ID":"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96","Type":"ContainerStarted","Data":"7d7f020e082032ee1889b99f405894c3b1b23b7cab5c8f52e84bef7fb7ff3f91"} Nov 03 02:17:46.443262 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:46.443178 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" event={"ID":"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96","Type":"ContainerStarted","Data":"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1"} Nov 03 02:17:46.443800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:46.443333 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:46.613476 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:46.613367 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" podStartSLOduration=1.603624905 podStartE2EDuration="2.613352914s" totalImagesPullingTime="1.009728009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:46.611422737 +0000 UTC m=+3602.190245324" watchObservedRunningTime="2026-11-03 02:17:46.613352914 +0000 UTC m=+3602.192175500" Nov 03 02:17:52.457801 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:52.457766 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:17:54.522755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.522723 4992 generic.go:356] "Generic (PLEG): container finished" podID="98d5d275-c7f7-499a-9f63-764451a91487" containerID="36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a" exitCode=0 Nov 03 02:17:54.523208 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.522799 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerDied","Data":"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a"} Nov 03 02:17:54.524036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.523957 4992 scope.go:122] "RemoveContainer" containerID="36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a" Nov 03 02:17:54.530045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.527539 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:54.530045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.527750 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:54.530045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:54.527849 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:17:55.531024 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:55.530987 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerStarted","Data":"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42"} Nov 03 02:17:56.047831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:56.047789 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:17:56.048048 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:56.048019 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:17:56.048136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:56.048081 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:17:57.048828 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:57.048791 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:17:57.049215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:57.048867 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:17:58.048489 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:58.048450 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:17:58.048694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:58.048522 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:17:59.048085 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:59.048042 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:17:59.048520 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:17:59.048124 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:18:00.049227 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:00.049186 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 02:18:00.049764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:00.049259 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 02:18:01.052241 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:01.052202 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:18:01.053027 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:01.053009 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:18:01.053865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:01.053821 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:18:02.152022 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:18:02.151983 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:18:28.759946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.759911 4992 generic.go:356] "Generic (PLEG): container finished" podID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerID="de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283" exitCode=0 Nov 03 02:18:28.760398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.759965 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerDied","Data":"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283"} Nov 03 02:18:28.760398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.760367 4992 scope.go:122] "RemoveContainer" containerID="de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283" Nov 03 02:18:28.762182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.762150 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:28.762273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.762229 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:28.762273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:28.762262 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:18:29.790340 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:29.790298 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerStarted","Data":"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844"} Nov 03 02:18:29.886668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:29.886626 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:18:29.888361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:29.888327 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:29.888524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:29.888396 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:30.876084 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:30.876037 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:30.876762 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:30.876729 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:31.886783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:31.886740 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:31.887316 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:31.886816 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:32.151833 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:18:32.151744 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:18:32.882441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:32.881998 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:32.882441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:32.882069 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:33.881584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:33.880982 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:33.881584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:33.881067 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:33.916596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:33.916538 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:18:34.878778 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:34.878737 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 02:18:34.879088 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:34.879056 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 02:18:35.877941 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:35.877903 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:18:35.879002 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:18:35.878982 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:19:00.051315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:00.051276 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2"] Nov 03 02:19:00.102846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:00.102802 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43616389-zrdf2"] Nov 03 02:19:00.596192 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:00.596155 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913c53c5-6fb4-4fb0-898e-93313fd11d32" path="/var/lib/kubelet/pods/913c53c5-6fb4-4fb0-898e-93313fd11d32/volumes" Nov 03 02:19:02.153239 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:19:02.153197 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:19:02.577980 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.577929 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:19:02.583517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.583485 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.638766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.638729 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3fb07e-openshift-default-dockercfg-whdsj\"" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672715 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brz79\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672783 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672812 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672884 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672919 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.672950 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.673011 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.673046 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.673687 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.673086 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.695073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.695030 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775061 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brz79\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775137 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775171 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775240 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775274 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775308 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775373 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775409 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.778584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.775449 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.782579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.779344 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.782579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.780296 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.782579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.781293 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.782579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.781942 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.782579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.782069 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.805756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.805716 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.810642 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.810542 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.843455 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.843383 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.843643 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.843469 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brz79\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\") pod \"e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:02.903632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:02.903298 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:03.342704 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:19:03.342668 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50dbdcc_3f3e_4f33_bdb2_c5091025212c.slice/crio-c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e WatchSource:0}: Error finding container c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e: Status 404 returned error can't find the container with id c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e Nov 03 02:19:03.345530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:03.345503 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:19:03.346165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:03.346132 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:03.346213 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:03.346198 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:03.346253 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:03.346231 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:04.019602 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:04.019528 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerStarted","Data":"d4f067c86a8bddecf591ef599672c7cdae7499d322c00ac2bcdf89fa8105ffaa"} Nov 03 02:19:04.019602 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:04.019607 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerStarted","Data":"c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e"} Nov 03 02:19:04.249867 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:04.249800 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podStartSLOduration=2.249778553 podStartE2EDuration="2.249778553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:04.195209075 +0000 UTC m=+3679.774031663" watchObservedRunningTime="2026-11-03 02:19:04.249778553 +0000 UTC m=+3679.828601141" Nov 03 02:19:04.904065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:04.904032 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:04.910681 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:04.910651 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:05.025831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:05.025802 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:05.026920 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:05.026902 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:19:11.054065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:11.054022 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc"] Nov 03 02:19:11.103446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:11.103413 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd5f318-2kkjc"] Nov 03 02:19:12.595657 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:12.595618 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0874cb-1153-438a-a0b6-4c72d28e2ede" path="/var/lib/kubelet/pods/ea0874cb-1153-438a-a0b6-4c72d28e2ede/volumes" Nov 03 02:19:24.781004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.780962 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:19:24.785144 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.785123 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.813955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.813913 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.813955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.813952 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814062 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814147 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814229 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvtn\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814276 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814266 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814493 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814351 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814493 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814391 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.814493 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.814408 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.839773 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.839730 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4f87d1-openshift-default-dockercfg-mk9lx\"" Nov 03 02:19:24.896694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.896655 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:19:24.916118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916080 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvtn\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916131 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916213 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916268 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916295 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916369 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916395 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916452 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916529 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916654 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.916868 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.916759 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.917060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.917036 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.917129 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.917058 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.917182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.917167 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.919029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.919008 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.919233 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.919214 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.979509 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.979469 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:24.979666 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:24.979554 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvtn\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:25.100247 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:25.100163 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:25.528842 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:19:25.528799 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bfae66_144a_499a_8cf1_706c02ea11f1.slice/crio-1f7ce125343292b029e96d29838277cb9ca7148056a6029aefb52691d96a07bb WatchSource:0}: Error finding container 1f7ce125343292b029e96d29838277cb9ca7148056a6029aefb52691d96a07bb: Status 404 returned error can't find the container with id 1f7ce125343292b029e96d29838277cb9ca7148056a6029aefb52691d96a07bb Nov 03 02:19:25.531753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:25.531712 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:25.531858 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:25.531773 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:19:25.531858 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:25.531795 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:25.531858 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:25.531831 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:19:26.208281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:26.208240 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" event={"ID":"79bfae66-144a-499a-8cf1-706c02ea11f1","Type":"ContainerStarted","Data":"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7"} Nov 03 02:19:26.208723 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:26.208297 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" event={"ID":"79bfae66-144a-499a-8cf1-706c02ea11f1","Type":"ContainerStarted","Data":"1f7ce125343292b029e96d29838277cb9ca7148056a6029aefb52691d96a07bb"} Nov 03 02:19:26.343432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:26.343378 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podStartSLOduration=2.343362937 podStartE2EDuration="2.343362937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:26.341456644 +0000 UTC m=+3701.920279232" watchObservedRunningTime="2026-11-03 02:19:26.343362937 +0000 UTC m=+3701.922185571" Nov 03 02:19:27.100381 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:27.100339 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:27.101878 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:27.101848 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 02:19:27.102014 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:27.101906 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 02:19:28.101447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:28.101407 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 02:19:28.102045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:28.101469 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 02:19:29.101418 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:29.101381 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 02:19:29.101624 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:29.101440 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 02:19:30.101611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:30.101571 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 02:19:30.102118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:30.101629 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 02:19:31.101611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:31.101534 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 03 02:19:31.101815 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:31.101633 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 03 02:19:32.105838 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:32.105798 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:32.106247 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:32.106235 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:32.107071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:32.107048 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:19:32.153503 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:19:32.153467 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:19:48.050665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:48.050626 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb"] Nov 03 02:19:48.102404 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:48.102364 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee74161d-c7ntb"] Nov 03 02:19:48.596708 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:48.596667 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e253e5-fbc1-40dc-877c-58d37ab03f29" path="/var/lib/kubelet/pods/29e253e5-fbc1-40dc-877c-58d37ab03f29/volumes" Nov 03 02:19:59.049856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:59.049772 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654"] Nov 03 02:19:59.101467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:19:59.101425 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bb8a9ad-kg654"] Nov 03 02:20:00.602758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:00.602716 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce8f1d0-7598-4ee8-a417-1280495c5086" path="/var/lib/kubelet/pods/4ce8f1d0-7598-4ee8-a417-1280495c5086/volumes" Nov 03 02:20:02.151660 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:20:02.151610 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:20:24.050955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:24.050916 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w"] Nov 03 02:20:24.102441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:24.102402 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93626c15-vmr6w"] Nov 03 02:20:24.595781 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:24.595745 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97139aa-4476-47ae-ab97-98ba71f415db" path="/var/lib/kubelet/pods/e97139aa-4476-47ae-ab97-98ba71f415db/volumes" Nov 03 02:20:32.152416 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:20:32.152376 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:20:35.051309 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:35.051271 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg"] Nov 03 02:20:35.102533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:35.102490 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caccfcd2-wqfrg"] Nov 03 02:20:36.595666 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:36.595629 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fa0cc9-37e1-4589-b4be-48f718e028fa" path="/var/lib/kubelet/pods/e4fa0cc9-37e1-4589-b4be-48f718e028fa/volumes" Nov 03 02:20:45.571849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:45.571812 4992 scope.go:122] "RemoveContainer" containerID="f8d588ae114959c67b3d19d61d24b439fad2dcdd8427c3b2e15925f8f2f380b5" Nov 03 02:20:45.583329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:45.583305 4992 scope.go:122] "RemoveContainer" containerID="19c9f13931189548df979337ff21dec87333f67a7af1150694994aee2581a411" Nov 03 02:20:45.594445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:45.594424 4992 scope.go:122] "RemoveContainer" containerID="6f5c7673896bd9a9ccdf3960cae3d8e66bf5ddcd2d44fa28da8bf64b094c08bd" Nov 03 02:20:53.916787 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:53.916753 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj"] Nov 03 02:20:53.920885 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:53.920868 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:53.974718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:53.974680 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj"] Nov 03 02:20:54.023341 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.023310 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:20:54.023525 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.023352 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:20:54.081798 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.081756 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv2wq\" (UniqueName: \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.081974 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.081940 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.082082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.082064 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.183188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.183099 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.183344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.183240 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.183436 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.183413 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv2wq\" (UniqueName: \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.183979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.183960 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.185522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.185505 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.246242 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.246206 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv2wq\" (UniqueName: \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\") pod \"maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.534182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.534105 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:54.911225 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:20:54.911189 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7241bf3_6d5e_4f28_a8f8_52c0accd1a5e.slice/crio-6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078 WatchSource:0}: Error finding container 6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078: Status 404 returned error can't find the container with id 6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078 Nov 03 02:20:54.913927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:54.913901 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj"] Nov 03 02:20:55.793997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:55.793961 4992 generic.go:356] "Generic (PLEG): container finished" podID="d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" containerID="46e7b03d7ec868f188b0858bb82f07b1abc78252fb19e87147041ca1f17c24b1" exitCode=0 Nov 03 02:20:55.794409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:55.794036 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" event={"ID":"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e","Type":"ContainerDied","Data":"46e7b03d7ec868f188b0858bb82f07b1abc78252fb19e87147041ca1f17c24b1"} Nov 03 02:20:55.794409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:55.794088 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" event={"ID":"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e","Type":"ContainerStarted","Data":"6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078"} Nov 03 02:20:56.959817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:56.959783 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:57.029230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.029194 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\") pod \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " Nov 03 02:20:57.029408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.029287 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\") pod \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " Nov 03 02:20:57.029408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.029401 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\" (UniqueName: \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\") pod \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\" (UID: \"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e\") " Nov 03 02:20:57.029702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.029669 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca" pod "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" (UID: "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:20:57.029927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.029905 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:57.031537 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.031509 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca" pod "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" (UID: "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:20:57.031656 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.031612 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq" pod "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" (UID: "d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e"). InnerVolumeSpecName "kube-api-access-tv2wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:20:57.130797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.130707 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:57.130797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.130748 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv2wq\" (UniqueName: \"kubernetes.io/projected/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e-kube-api-access-tv2wq\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:20:57.808969 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.808935 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" Nov 03 02:20:57.809147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.808970 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj" event={"ID":"d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e","Type":"ContainerDied","Data":"6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078"} Nov 03 02:20:57.809147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:20:57.809021 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e0c8fc6256d9570aeeacc21a58bc48f41abdd8005f9256140f7b0a6289f6078" Nov 03 02:21:02.151854 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:21:02.151802 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:21:05.824967 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.824930 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj"] Nov 03 02:21:05.825466 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.825453 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" containerName="revoke-keys" Nov 03 02:21:05.825509 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.825467 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" containerName="revoke-keys" Nov 03 02:21:05.825608 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.825598 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" containerName="revoke-keys" Nov 03 02:21:05.829099 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.829080 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:05.884399 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.884360 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj"] Nov 03 02:21:05.931527 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.931492 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:21:05.931527 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.931500 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:21:05.938189 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.938157 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98vv\" (UniqueName: \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:05.938377 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.938306 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:05.938434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:05.938364 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.039237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.039195 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.039438 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.039275 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.039438 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.039374 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98vv\" (UniqueName: \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.040058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.040035 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.041721 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.041702 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.102799 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.102727 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98vv\" (UniqueName: \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\") pod \"maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.142927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.142890 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:06.516619 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:21:06.516543 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7781e0d7_faab_48c3_8615_4d205142d25c.slice/crio-2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69 WatchSource:0}: Error finding container 2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69: Status 404 returned error can't find the container with id 2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69 Nov 03 02:21:06.519502 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.519448 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj"] Nov 03 02:21:06.865923 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.865886 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" event={"ID":"7781e0d7-faab-48c3-8615-4d205142d25c","Type":"ContainerStarted","Data":"9d71d70fb0e9a9921b4b447378a4b4c00bd50fb7edb00fc0c25e66bedcb8205a"} Nov 03 02:21:06.866322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.865941 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" event={"ID":"7781e0d7-faab-48c3-8615-4d205142d25c","Type":"ContainerStarted","Data":"2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69"} Nov 03 02:21:06.985365 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:06.985310 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" podStartSLOduration=1.9852969219999999 podStartE2EDuration="1.985296922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:21:06.984115039 +0000 UTC m=+3802.562937625" watchObservedRunningTime="2026-11-03 02:21:06.985296922 +0000 UTC m=+3802.564119514" Nov 03 02:21:07.874677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:07.874642 4992 generic.go:356] "Generic (PLEG): container finished" podID="7781e0d7-faab-48c3-8615-4d205142d25c" containerID="9d71d70fb0e9a9921b4b447378a4b4c00bd50fb7edb00fc0c25e66bedcb8205a" exitCode=0 Nov 03 02:21:07.875132 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:07.874723 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" event={"ID":"7781e0d7-faab-48c3-8615-4d205142d25c","Type":"ContainerDied","Data":"9d71d70fb0e9a9921b4b447378a4b4c00bd50fb7edb00fc0c25e66bedcb8205a"} Nov 03 02:21:09.040971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.040936 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:09.176613 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.176547 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\" (UniqueName: \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\") pod \"7781e0d7-faab-48c3-8615-4d205142d25c\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " Nov 03 02:21:09.177200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.177165 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\") pod \"7781e0d7-faab-48c3-8615-4d205142d25c\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " Nov 03 02:21:09.177472 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.177447 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\") pod \"7781e0d7-faab-48c3-8615-4d205142d25c\" (UID: \"7781e0d7-faab-48c3-8615-4d205142d25c\") " Nov 03 02:21:09.179075 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.179042 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca" pod "7781e0d7-faab-48c3-8615-4d205142d25c" (UID: "7781e0d7-faab-48c3-8615-4d205142d25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:21:09.180064 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.180029 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv" pod "7781e0d7-faab-48c3-8615-4d205142d25c" (UID: "7781e0d7-faab-48c3-8615-4d205142d25c"). InnerVolumeSpecName "kube-api-access-q98vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:21:09.182653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.181662 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca" pod "7781e0d7-faab-48c3-8615-4d205142d25c" (UID: "7781e0d7-faab-48c3-8615-4d205142d25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:21:09.280386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.280337 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:09.280386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.280382 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7781e0d7-faab-48c3-8615-4d205142d25c-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:09.280386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.280394 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q98vv\" (UniqueName: \"kubernetes.io/projected/7781e0d7-faab-48c3-8615-4d205142d25c-kube-api-access-q98vv\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:09.889927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.889893 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" event={"ID":"7781e0d7-faab-48c3-8615-4d205142d25c","Type":"ContainerDied","Data":"2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69"} Nov 03 02:21:09.890141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.889948 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2565bc83d727d35027c61a156361577ec35b7858792c21be293e4ac62fd8bd69" Nov 03 02:21:09.890141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:09.889900 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj" Nov 03 02:21:32.151463 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:21:32.151421 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:21:42.096629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.096594 4992 generic.go:356] "Generic (PLEG): container finished" podID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerID="d4f067c86a8bddecf591ef599672c7cdae7499d322c00ac2bcdf89fa8105ffaa" exitCode=0 Nov 03 02:21:42.097071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.096666 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerDied","Data":"d4f067c86a8bddecf591ef599672c7cdae7499d322c00ac2bcdf89fa8105ffaa"} Nov 03 02:21:42.097134 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.097095 4992 scope.go:122] "RemoveContainer" containerID="d4f067c86a8bddecf591ef599672c7cdae7499d322c00ac2bcdf89fa8105ffaa" Nov 03 02:21:42.098845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.098815 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:42.098906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.098890 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:42.098962 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:42.098922 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:21:43.104831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:43.104795 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerStarted","Data":"a42cd4590b44dc64e8936f62668ea94f6d76a7072872c72b3793faf1f059173e"} Nov 03 02:21:43.904635 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:43.904599 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:21:43.904827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:43.904783 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:21:43.904873 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:43.904841 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 02:21:44.904684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:44.904652 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:21:44.905170 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:44.905143 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 02:21:45.628086 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:45.628056 4992 scope.go:122] "RemoveContainer" containerID="2fbadcbc6b4a677f7a1a435e148a4047aa2d9f15fe0264b52bf68a5bb3235509" Nov 03 02:21:45.638115 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:45.638093 4992 scope.go:122] "RemoveContainer" containerID="b6b8060abe0a07f790c7b5054ae4009fce42b79d730e73fa98988f40593ed097" Nov 03 02:21:45.648000 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:45.647973 4992 scope.go:122] "RemoveContainer" containerID="d2afa483cf2606c7a398a4783c826fdda1f4a331f1dc288f2907448cfc0b579d" Nov 03 02:21:45.904808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:45.904726 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:21:45.904808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:45.904783 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 02:21:46.904574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:46.904525 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:21:46.904765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:46.904614 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 02:21:47.592092 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:47.592055 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:21:47.592496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:47.592283 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" podUID="b357d84b-022b-42d3-b485-4407078db4d9" containerName="maas-api" containerID="cri-o://e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16" gracePeriod=30 Nov 03 02:21:47.909754 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:47.908542 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 02:21:47.909754 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:47.908624 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 02:21:48.043172 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.043145 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:21:48.141707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.141655 4992 generic.go:356] "Generic (PLEG): container finished" podID="b357d84b-022b-42d3-b485-4407078db4d9" containerID="e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16" exitCode=0 Nov 03 02:21:48.141707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.141706 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" Nov 03 02:21:48.142010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.141752 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" event={"ID":"b357d84b-022b-42d3-b485-4407078db4d9","Type":"ContainerDied","Data":"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16"} Nov 03 02:21:48.142010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.141813 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p" event={"ID":"b357d84b-022b-42d3-b485-4407078db4d9","Type":"ContainerDied","Data":"c4b8e30d2e7c43e637b87bed5a9a49f0440353dcda832ab1dbbe31bb03c1b757"} Nov 03 02:21:48.142010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.141845 4992 scope.go:122] "RemoveContainer" containerID="e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16" Nov 03 02:21:48.152033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.152002 4992 scope.go:122] "RemoveContainer" containerID="e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16" Nov 03 02:21:48.152343 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:21:48.152321 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16\": container with ID starting with e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16 not found: ID does not exist" containerID="e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16" Nov 03 02:21:48.152494 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.152353 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16"} err="failed to get container status \"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16\": rpc error: code = NotFound desc = could not find container \"e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16\": container with ID starting with e8502e482bbf944e0372b71ed0528023fc1941eb03c1851894d825d769c67e16 not found: ID does not exist" Nov 03 02:21:48.211533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.211498 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") pod \"b357d84b-022b-42d3-b485-4407078db4d9\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " Nov 03 02:21:48.211724 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.211699 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\" (UniqueName: \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\") pod \"b357d84b-022b-42d3-b485-4407078db4d9\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " Nov 03 02:21:48.211811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.211758 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\") pod \"b357d84b-022b-42d3-b485-4407078db4d9\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " Nov 03 02:21:48.211811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.211789 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\") pod \"b357d84b-022b-42d3-b485-4407078db4d9\" (UID: \"b357d84b-022b-42d3-b485-4407078db4d9\") " Nov 03 02:21:48.212256 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.212218 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca" pod "b357d84b-022b-42d3-b485-4407078db4d9" (UID: "b357d84b-022b-42d3-b485-4407078db4d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:21:48.212414 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.212388 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b357d84b-022b-42d3-b485-4407078db4d9-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:48.213788 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.213759 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls" pod "b357d84b-022b-42d3-b485-4407078db4d9" (UID: "b357d84b-022b-42d3-b485-4407078db4d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:21:48.213915 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.213892 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk" pod "b357d84b-022b-42d3-b485-4407078db4d9" (UID: "b357d84b-022b-42d3-b485-4407078db4d9"). InnerVolumeSpecName "kube-api-access-qhslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:21:48.214061 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.214043 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs" pod "b357d84b-022b-42d3-b485-4407078db4d9" (UID: "b357d84b-022b-42d3-b485-4407078db4d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:21:48.314093 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.314052 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:48.314093 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.314088 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhslk\" (UniqueName: \"kubernetes.io/projected/b357d84b-022b-42d3-b485-4407078db4d9-kube-api-access-qhslk\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:48.314093 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.314099 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b357d84b-022b-42d3-b485-4407078db4d9-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:21:48.567772 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.567682 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:21:48.619438 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.619401 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c70365-655d4846dd-fc55p"] Nov 03 02:21:48.910346 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.910264 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:21:48.910701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.910680 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:21:48.911465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:48.911446 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:21:50.596354 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:21:50.596320 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b357d84b-022b-42d3-b485-4407078db4d9" path="/var/lib/kubelet/pods/b357d84b-022b-42d3-b485-4407078db4d9/volumes" Nov 03 02:22:02.152276 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:22:02.152236 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:22:28.930191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:28.930130 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:22:28.931082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:28.930391 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" podUID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" containerName="maas-api" containerID="cri-o://feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1" gracePeriod=30 Nov 03 02:22:29.240094 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.240062 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:22:29.362823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.362772 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\") pod \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " Nov 03 02:22:29.363042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.362879 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\" (UniqueName: \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\") pod \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " Nov 03 02:22:29.363042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.362975 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\") pod \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " Nov 03 02:22:29.363172 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.363041 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\") pod \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\" (UID: \"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96\") " Nov 03 02:22:29.366260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.366185 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca" pod "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" (UID: "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:29.384592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.378754 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls" pod "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" (UID: "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:29.384592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.379922 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs" pod "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" (UID: "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:22:29.384592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.381794 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5" pod "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" (UID: "2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96"). InnerVolumeSpecName "kube-api-access-xtvg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:29.412970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.412940 4992 generic.go:356] "Generic (PLEG): container finished" podID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" containerID="feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1" exitCode=0 Nov 03 02:22:29.413136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.413009 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" Nov 03 02:22:29.413136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.413035 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" event={"ID":"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96","Type":"ContainerDied","Data":"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1"} Nov 03 02:22:29.413136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.413104 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch" event={"ID":"2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96","Type":"ContainerDied","Data":"7d7f020e082032ee1889b99f405894c3b1b23b7cab5c8f52e84bef7fb7ff3f91"} Nov 03 02:22:29.413136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.413134 4992 scope.go:122] "RemoveContainer" containerID="feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1" Nov 03 02:22:29.425569 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.425528 4992 scope.go:122] "RemoveContainer" containerID="feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1" Nov 03 02:22:29.425879 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:22:29.425861 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1\": container with ID starting with feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1 not found: ID does not exist" containerID="feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1" Nov 03 02:22:29.425942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.425888 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1"} err="failed to get container status \"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1\": rpc error: code = NotFound desc = could not find container \"feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1\": container with ID starting with feed96cc9954933a52cb20a3f4fd163741117982e4474693ef1bc5e587111be1 not found: ID does not exist" Nov 03 02:22:29.464800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.464710 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:29.464800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.464745 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvg5\" (UniqueName: \"kubernetes.io/projected/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-kube-api-access-xtvg5\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:29.464800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.464762 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:29.464800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.464776 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:29.584448 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.584405 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:22:29.588677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:29.588643 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-50ca6d-7c5699bb67-jrnch"] Nov 03 02:22:30.597603 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:30.597539 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" path="/var/lib/kubelet/pods/2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96/volumes" Nov 03 02:22:30.943467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:30.943426 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj"] Nov 03 02:22:30.996707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:30.996673 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-50ca6d-6g9zj"] Nov 03 02:22:32.152505 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:22:32.152466 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:22:32.595896 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:32.595860 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e" path="/var/lib/kubelet/pods/d7241bf3-6d5e-4f28-a8f8-52c0accd1a5e/volumes" Nov 03 02:22:42.755583 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:42.740043 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj"] Nov 03 02:22:42.755583 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:42.743817 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d8cc55-6xhgj"] Nov 03 02:22:43.002955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:43.002904 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:22:43.003430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:43.003395 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" containerID="cri-o://51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844" gracePeriod=30 Nov 03 02:22:44.595449 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:44.595414 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7781e0d7-faab-48c3-8615-4d205142d25c" path="/var/lib/kubelet/pods/7781e0d7-faab-48c3-8615-4d205142d25c/volumes" Nov 03 02:22:45.165522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:45.165488 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:22:45.167419 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:45.167392 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:22:48.295457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.295430 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:22:48.429957 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.429921 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.429975 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430011 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430046 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430090 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430154 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430187 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430238 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430380 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\") pod \"663cf534-2777-4e07-ae36-bf7c7464dbb2\" (UID: \"663cf534-2777-4e07-ae36-bf7c7464dbb2\") " Nov 03 02:22:48.430430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430395 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:48.430700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430444 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:22:48.430700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430452 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:48.430806 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430739 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:48.430806 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.430770 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:48.431114 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.431087 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.431114 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.431114 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.431322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.431130 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/663cf534-2777-4e07-ae36-bf7c7464dbb2-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.431322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.431144 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.431322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.431157 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.432508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.432476 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:22:48.432677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.432526 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:22:48.432677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.432605 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "kube-api-access-pjkvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:48.432677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.432642 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token" pod "663cf534-2777-4e07-ae36-bf7c7464dbb2" (UID: "663cf534-2777-4e07-ae36-bf7c7464dbb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:22:48.532148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.532064 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.532148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.532095 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.532148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.532105 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.532148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.532113 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkvf\" (UniqueName: \"kubernetes.io/projected/663cf534-2777-4e07-ae36-bf7c7464dbb2-kube-api-access-pjkvf\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:22:48.545606 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.545571 4992 generic.go:356] "Generic (PLEG): container finished" podID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerID="51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844" exitCode=0 Nov 03 02:22:48.545770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.545626 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" Nov 03 02:22:48.545770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.545646 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerDied","Data":"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844"} Nov 03 02:22:48.545770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.545719 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v" event={"ID":"663cf534-2777-4e07-ae36-bf7c7464dbb2","Type":"ContainerDied","Data":"90ac22529f1b8b5777161e1f7cf7f73fe44a69148c315354e814aa24147e51b8"} Nov 03 02:22:48.545770 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.545762 4992 scope.go:122] "RemoveContainer" containerID="51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844" Nov 03 02:22:48.556784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.556763 4992 scope.go:122] "RemoveContainer" containerID="de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283" Nov 03 02:22:48.566191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.566169 4992 scope.go:122] "RemoveContainer" containerID="51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844" Nov 03 02:22:48.566458 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:22:48.566437 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844\": container with ID starting with 51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844 not found: ID does not exist" containerID="51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844" Nov 03 02:22:48.566611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.566467 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844"} err="failed to get container status \"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844\": rpc error: code = NotFound desc = could not find container \"51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844\": container with ID starting with 51ef0be42af703d8ed183885b7857bfd6efece40c93a60b0032f8d58758fa844 not found: ID does not exist" Nov 03 02:22:48.566611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.566487 4992 scope.go:122] "RemoveContainer" containerID="de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283" Nov 03 02:22:48.566792 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:22:48.566770 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283\": container with ID starting with de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283 not found: ID does not exist" containerID="de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283" Nov 03 02:22:48.566837 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.566802 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283"} err="failed to get container status \"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283\": rpc error: code = NotFound desc = could not find container \"de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283\": container with ID starting with de53f05b1e5278a968dd81033ff27ba186e035b9676e4085e8d0942ffefad283 not found: ID does not exist" Nov 03 02:22:48.718711 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.718672 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:22:48.725016 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:48.724980 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8cc55-openshift-default-56c5c46c89-84q9v"] Nov 03 02:22:50.594755 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:22:50.594724 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" path="/var/lib/kubelet/pods/663cf534-2777-4e07-ae36-bf7c7464dbb2/volumes" Nov 03 02:23:02.152917 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:23:02.152827 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:23:29.487866 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:29.487823 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:23:29.488308 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:29.488122 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" containerID="cri-o://62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42" gracePeriod=30 Nov 03 02:23:31.056259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:31.056209 4992 patch_prober.go:29] interesting pod/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:23:31.056705 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:31.056298 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:23:31.135519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:31.135485 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:23:31.135745 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:31.135717 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" containerID="cri-o://38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7" gracePeriod=30 Nov 03 02:23:32.108274 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:32.108236 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:23:32.108680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:32.108304 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:23:32.151441 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:23:32.151392 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:23:34.772090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.772062 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:23:34.817124 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817085 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817298 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817144 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817340 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817317 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817364 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817424 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817453 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817436 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:34.817592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817454 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817486 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817517 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817613 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:34.817783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817636 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\") pod \"98d5d275-c7f7-499a-9f63-764451a91487\" (UID: \"98d5d275-c7f7-499a-9f63-764451a91487\") " Nov 03 02:23:34.817783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817720 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:34.817783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817752 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:34.818003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.817955 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:34.818284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.818231 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.818284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.818256 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.818284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.818272 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.818456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.818288 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d5d275-c7f7-499a-9f63-764451a91487-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.818456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.818301 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.819881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.819847 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:34.819997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.819946 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "kube-api-access-xjw7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:34.820091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.820069 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:23:34.820659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.820635 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy" pod "98d5d275-c7f7-499a-9f63-764451a91487" (UID: "98d5d275-c7f7-499a-9f63-764451a91487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:34.859313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.859274 4992 generic.go:356] "Generic (PLEG): container finished" podID="98d5d275-c7f7-499a-9f63-764451a91487" containerID="62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42" exitCode=0 Nov 03 02:23:34.859505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.859328 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" Nov 03 02:23:34.859505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.859351 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerDied","Data":"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42"} Nov 03 02:23:34.859505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.859405 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt" event={"ID":"98d5d275-c7f7-499a-9f63-764451a91487","Type":"ContainerDied","Data":"12c2472d26b9b3ff97ec9922a7195073630a8d653226641d5bf98747497274da"} Nov 03 02:23:34.859505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.859430 4992 scope.go:122] "RemoveContainer" containerID="62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42" Nov 03 02:23:34.870429 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.870406 4992 scope.go:122] "RemoveContainer" containerID="36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a" Nov 03 02:23:34.880312 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.880291 4992 scope.go:122] "RemoveContainer" containerID="62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42" Nov 03 02:23:34.880615 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:23:34.880595 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42\": container with ID starting with 62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42 not found: ID does not exist" containerID="62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42" Nov 03 02:23:34.880670 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.880625 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42"} err="failed to get container status \"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42\": rpc error: code = NotFound desc = could not find container \"62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42\": container with ID starting with 62325105be6cbdab748a1b4c7cbcdad5c15737315d1cb9ba7551a5534dd95e42 not found: ID does not exist" Nov 03 02:23:34.880670 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.880641 4992 scope.go:122] "RemoveContainer" containerID="36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a" Nov 03 02:23:34.880853 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:23:34.880835 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a\": container with ID starting with 36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a not found: ID does not exist" containerID="36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a" Nov 03 02:23:34.880893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.880859 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a"} err="failed to get container status \"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a\": rpc error: code = NotFound desc = could not find container \"36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a\": container with ID starting with 36ce3a5109e34dfd571a6ae56b3e6136802a4aea72ff83127a7b747d8a1f996a not found: ID does not exist" Nov 03 02:23:34.919259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.919221 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d5d275-c7f7-499a-9f63-764451a91487-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.919259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.919255 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw7w\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-kube-api-access-xjw7w\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.919259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.919269 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d5d275-c7f7-499a-9f63-764451a91487-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:34.919504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:34.919278 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d5d275-c7f7-499a-9f63-764451a91487-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:35.074069 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:35.073982 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:23:35.078358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:35.078332 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-aab75e-openshift-default-5c9474977c-wpkbt"] Nov 03 02:23:36.292078 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292042 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:23:36.292549 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292534 4992 cpu_manager.go:414] "containerMap: removing container" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.292629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292550 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.292629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292580 4992 cpu_manager.go:414] "containerMap: removing container" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.292629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292590 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.292629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292612 4992 cpu_manager.go:414] "containerMap: removing container" podUID="7781e0d7-faab-48c3-8615-4d205142d25c" containerName="revoke-keys" Nov 03 02:23:36.292629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292617 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7781e0d7-faab-48c3-8615-4d205142d25c" containerName="revoke-keys" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292649 4992 cpu_manager.go:414] "containerMap: removing container" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292654 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292661 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b357d84b-022b-42d3-b485-4407078db4d9" containerName="maas-api" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292665 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b357d84b-022b-42d3-b485-4407078db4d9" containerName="maas-api" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292674 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" containerName="maas-api" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292677 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" containerName="maas-api" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292686 4992 cpu_manager.go:414] "containerMap: removing container" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292691 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.292786 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292784 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292793 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b357d84b-022b-42d3-b485-4407078db4d9" containerName="maas-api" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292799 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2661e6c5-9fff-40fd-9f47-bcc7b6ba0a96" containerName="maas-api" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292805 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d5d275-c7f7-499a-9f63-764451a91487" containerName="istio-proxy" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292813 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292819 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="663cf534-2777-4e07-ae36-bf7c7464dbb2" containerName="istio-proxy" Nov 03 02:23:36.293047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.292825 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="7781e0d7-faab-48c3-8615-4d205142d25c" containerName="revoke-keys" Nov 03 02:23:36.296329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.296304 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336603 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336725 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336767 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336794 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336834 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336951 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf47n\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.336979 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.337010 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.337571 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.337043 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.361579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.359123 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e051f3-openshift-default-dockercfg-klmx2\"" Nov 03 02:23:36.392454 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.392414 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:23:36.456928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.446344 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf47n\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.458248 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.458216 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.458498 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.458477 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.459188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459163 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.459396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459376 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.459636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459616 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.459795 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459777 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.459926 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459910 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.460068 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.460051 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.460188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.460157 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.460252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.459932 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.460674 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.460645 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.460994 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.460971 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.461318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.461293 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.465664 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.465621 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.466993 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.466936 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.493018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.492989 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:23:36.519752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.519719 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf47n\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.522149 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.522119 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\") pod \"e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.561614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.561576 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.561614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.561617 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.561848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.561731 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.561939 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.561878 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.561987 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562107 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562089 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562141 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:23:36.562185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562156 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562239 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562143 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.562410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562349 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.562410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562366 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\") pod \"79bfae66-144a-499a-8cf1-706c02ea11f1\" (UID: \"79bfae66-144a-499a-8cf1-706c02ea11f1\") " Nov 03 02:23:36.562515 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562366 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.562601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.562571 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.563058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.563035 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.563174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.563061 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.563174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.563076 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.563174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.563090 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bfae66-144a-499a-8cf1-706c02ea11f1-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.563174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.563103 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.564333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.564306 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "kube-api-access-hqvtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:36.564504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.564479 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:23:36.564979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.564889 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:23:36.564979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.564942 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token" pod "79bfae66-144a-499a-8cf1-706c02ea11f1" (UID: "79bfae66-144a-499a-8cf1-706c02ea11f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:23:36.598848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.598807 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d5d275-c7f7-499a-9f63-764451a91487" path="/var/lib/kubelet/pods/98d5d275-c7f7-499a-9f63-764451a91487/volumes" Nov 03 02:23:36.617523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.617303 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:36.664517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.664486 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.664733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.664599 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvtn\" (UniqueName: \"kubernetes.io/projected/79bfae66-144a-499a-8cf1-706c02ea11f1-kube-api-access-hqvtn\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.664733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.664621 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.664733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.664638 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bfae66-144a-499a-8cf1-706c02ea11f1-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:23:36.877189 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.877157 4992 generic.go:356] "Generic (PLEG): container finished" podID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerID="38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7" exitCode=0 Nov 03 02:23:36.877372 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.877209 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" Nov 03 02:23:36.877372 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.877247 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" event={"ID":"79bfae66-144a-499a-8cf1-706c02ea11f1","Type":"ContainerDied","Data":"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7"} Nov 03 02:23:36.877372 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.877320 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp" event={"ID":"79bfae66-144a-499a-8cf1-706c02ea11f1","Type":"ContainerDied","Data":"1f7ce125343292b029e96d29838277cb9ca7148056a6029aefb52691d96a07bb"} Nov 03 02:23:36.877372 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.877355 4992 scope.go:122] "RemoveContainer" containerID="38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7" Nov 03 02:23:36.888020 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.887999 4992 scope.go:122] "RemoveContainer" containerID="38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7" Nov 03 02:23:36.888356 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:23:36.888325 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7\": container with ID starting with 38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7 not found: ID does not exist" containerID="38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7" Nov 03 02:23:36.888457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:36.888352 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7"} err="failed to get container status \"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7\": rpc error: code = NotFound desc = could not find container \"38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7\": container with ID starting with 38c583568c813ecdc790a4379430bf2786926d72c5bfdcdca226b124236fa5c7 not found: ID does not exist" Nov 03 02:23:37.050893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.050799 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:23:37.096346 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:23:37.096311 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f242d8f_e746_48ef_a383_9e06f3565481.slice/crio-5b6672cfb02b0c388284a2b2a8d8b67975c8aaf4178e670f6082394dd35c3e1c WatchSource:0}: Error finding container 5b6672cfb02b0c388284a2b2a8d8b67975c8aaf4178e670f6082394dd35c3e1c: Status 404 returned error can't find the container with id 5b6672cfb02b0c388284a2b2a8d8b67975c8aaf4178e670f6082394dd35c3e1c Nov 03 02:23:37.099421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.099377 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:37.099538 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.099464 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:37.099538 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.099510 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:37.105006 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.104971 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-rkcbp"] Nov 03 02:23:37.105118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.105022 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:23:37.888271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.888227 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerStarted","Data":"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218"} Nov 03 02:23:37.888697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:37.888289 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerStarted","Data":"5b6672cfb02b0c388284a2b2a8d8b67975c8aaf4178e670f6082394dd35c3e1c"} Nov 03 02:23:38.013076 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.013018 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podStartSLOduration=2.01300413 podStartE2EDuration="2.01300413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:38.011930819 +0000 UTC m=+3953.590753404" watchObservedRunningTime="2026-11-03 02:23:38.01300413 +0000 UTC m=+3953.591826715" Nov 03 02:23:38.595846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.595812 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" path="/var/lib/kubelet/pods/79bfae66-144a-499a-8cf1-706c02ea11f1/volumes" Nov 03 02:23:38.617504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.617469 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:38.623807 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.623782 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:38.895464 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.895386 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:38.896458 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:38.896430 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:23:49.942657 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:49.942612 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:23:49.943340 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:49.943317 4992 cpu_manager.go:414] "containerMap: removing container" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" Nov 03 02:23:49.943430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:49.943341 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" Nov 03 02:23:49.943551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:49.943538 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bfae66-144a-499a-8cf1-706c02ea11f1" containerName="istio-proxy" Nov 03 02:23:49.947058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:49.947034 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.001271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.001233 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\"" Nov 03 02:23:50.048347 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048300 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048362 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048506 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048714 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048553 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048711 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048749 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048792 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrtr\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048924 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048908 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.048986 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.048972 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.055954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.055926 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:23:50.150426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150383 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150686 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150491 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150686 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150528 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150686 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150664 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150701 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150783 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150817 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.150880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150861 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrtr\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.151111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.150916 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.151843 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.151815 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.166808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.152300 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.167048 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.152836 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.167146 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.153063 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.167224 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.153270 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.170019 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.169986 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.211421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.211351 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.231437 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.231382 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.233705 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.233599 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrtr\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-m8x8g\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.262363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.262324 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:50.723235 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.723186 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:23:50.772710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.772663 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:50.772838 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.772763 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:50.772838 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.772807 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:23:50.992323 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.992222 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" event={"ID":"09c89df1-b20c-444a-ba37-65b8ce50eae0","Type":"ContainerStarted","Data":"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734"} Nov 03 02:23:50.992323 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:50.992281 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" event={"ID":"09c89df1-b20c-444a-ba37-65b8ce50eae0","Type":"ContainerStarted","Data":"d69b2bf0cee7d9c1a046a58a0acb0fc061fa67a483091328e95502362eb04293"} Nov 03 02:23:51.117376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:51.117320 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podStartSLOduration=2.117303522 podStartE2EDuration="2.117303522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:51.115774859 +0000 UTC m=+3966.694597457" watchObservedRunningTime="2026-11-03 02:23:51.117303522 +0000 UTC m=+3966.696126107" Nov 03 02:23:51.262797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:51.262707 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:51.264460 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:51.264432 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:51.264601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:51.264492 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:52.282585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:52.279305 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:52.282585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:52.279402 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:53.279584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:53.275091 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:53.279584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:53.275156 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:54.277596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:54.269180 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:54.277596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:54.269244 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:55.286626 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:55.283984 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:55.286626 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:55.284067 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:56.279584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:56.277848 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:56.279584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:56.277916 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:57.263751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:57.263707 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:57.264192 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:57.263795 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:58.262984 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:58.262929 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 02:23:58.263178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:58.263008 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 02:23:59.407990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:59.407950 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:59.408504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:59.408459 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:23:59.409320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:23:59.409298 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:24:02.151193 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:24:02.151154 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:24:20.178522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.178484 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:24:20.183375 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.183349 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.277621 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.277582 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:24:20.320432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320396 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320440 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320540 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320635 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320698 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320815 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320735 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320815 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320793 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4jh\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320879 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320834 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.320954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.320895 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422119 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422067 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422149 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422204 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4jh\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422268 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422316 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422385 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422417 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422484 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422544 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422803 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422645 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422803 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422746 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.422918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422797 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.423023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.422997 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.423268 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.423241 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.424759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.424730 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.425122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.425103 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.485440 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.485368 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.485607 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.485477 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4jh\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4d5mq\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.499164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.499138 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:20.925342 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:24:20.925308 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094ee459_5323_4cd3_bf3e_db8e9a49f56c.slice/crio-ddf97f154a0d10b73e6228a28bcc80b7f20925d2898da2bd3cdfdaa7017198ea WatchSource:0}: Error finding container ddf97f154a0d10b73e6228a28bcc80b7f20925d2898da2bd3cdfdaa7017198ea: Status 404 returned error can't find the container with id ddf97f154a0d10b73e6228a28bcc80b7f20925d2898da2bd3cdfdaa7017198ea Nov 03 02:24:20.927856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.927830 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:24:20.928406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.928374 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:20.928508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.928461 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:20.928508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:20.928493 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:21.213877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.213772 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" event={"ID":"094ee459-5323-4cd3-bf3e-db8e9a49f56c","Type":"ContainerStarted","Data":"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee"} Nov 03 02:24:21.213877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.213841 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" event={"ID":"094ee459-5323-4cd3-bf3e-db8e9a49f56c","Type":"ContainerStarted","Data":"ddf97f154a0d10b73e6228a28bcc80b7f20925d2898da2bd3cdfdaa7017198ea"} Nov 03 02:24:21.340620 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.340550 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podStartSLOduration=1.340534449 podStartE2EDuration="1.340534449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:21.337657053 +0000 UTC m=+3996.916479635" watchObservedRunningTime="2026-11-03 02:24:21.340534449 +0000 UTC m=+3996.919357034" Nov 03 02:24:21.499542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.499449 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:21.501409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.501374 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:21.501600 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:21.501427 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:22.500045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:22.500010 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:22.500533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:22.500498 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:23.500342 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:23.500302 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:23.500815 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:23.500359 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:24.500392 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:24.500312 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:24.500392 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:24.500378 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:25.500402 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:25.500359 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:25.500953 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:25.500428 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:26.500361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:26.500317 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:26.500587 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:26.500378 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:27.499957 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:27.499916 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:27.500157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:27.499976 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:28.499825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:28.499787 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 02:24:28.500234 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:28.499858 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 02:24:29.504906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:29.504866 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:29.505320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:29.505211 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:29.506066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:29.506050 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:24:32.151507 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:24:32.151470 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:24:35.299952 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.299917 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:24:35.304400 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.304374 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326477 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326713 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326538 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326721 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326758 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326871 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326778 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326945 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326877 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.326992 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.326954 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.327072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.327053 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mw2t\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.327104 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.327083 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.413281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.413246 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:24:35.428280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428237 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428344 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428366 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428383 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428406 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428449 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428794 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428496 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mw2t\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428794 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428524 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428794 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428618 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.428960 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.428860 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.429366 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.429100 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.429366 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.429273 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.429554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.429530 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.430263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.430235 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.431061 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.431027 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.431599 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.431539 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.491831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.491790 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.491987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.491900 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mw2t\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-fqhg2\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:35.619363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:35.619270 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:36.047917 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:24:36.047881 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36371633_6ead_46ab_9bda_65014683a63a.slice/crio-2fa0043d2474184ca35d3b5af0edb85512f06637b2c695c1d979efe216534e81 WatchSource:0}: Error finding container 2fa0043d2474184ca35d3b5af0edb85512f06637b2c695c1d979efe216534e81: Status 404 returned error can't find the container with id 2fa0043d2474184ca35d3b5af0edb85512f06637b2c695c1d979efe216534e81 Nov 03 02:24:36.050716 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.050682 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:36.050802 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.050752 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:36.050802 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.050783 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:24:36.051064 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.051043 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:24:36.311462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.311368 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" event={"ID":"36371633-6ead-46ab-9bda-65014683a63a","Type":"ContainerStarted","Data":"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8"} Nov 03 02:24:36.311462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.311451 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" event={"ID":"36371633-6ead-46ab-9bda-65014683a63a","Type":"ContainerStarted","Data":"2fa0043d2474184ca35d3b5af0edb85512f06637b2c695c1d979efe216534e81"} Nov 03 02:24:36.451765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.451714 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podStartSLOduration=1.451700044 podStartE2EDuration="1.451700044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:36.450348502 +0000 UTC m=+4012.029171113" watchObservedRunningTime="2026-11-03 02:24:36.451700044 +0000 UTC m=+4012.030522631" Nov 03 02:24:36.620422 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:36.620335 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:37.621701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:37.621606 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:37.621701 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:37.621665 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:38.619662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:38.619627 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:38.619841 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:38.619680 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:39.619882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:39.619840 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:39.620302 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:39.619906 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:40.620216 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:40.620179 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:40.620744 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:40.620268 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:41.620040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:41.619998 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:41.620244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:41.620071 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:42.619834 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:42.619799 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:42.620018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:42.619861 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:43.619826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:43.619790 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 02:24:43.620240 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:43.619849 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 02:24:44.632309 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:44.632278 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:44.632771 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:44.632732 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:24:44.633507 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:24:44.633487 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:25:02.152294 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:02.152255 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:25:02.260825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:02.260780 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:25:02.261045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:02.260994 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" containerID="cri-o://a42cd4590b44dc64e8936f62668ea94f6d76a7072872c72b3793faf1f059173e" gracePeriod=30 Nov 03 02:25:03.648194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.648149 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:25:03.652872 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.652844 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707262 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707333 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707393 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707493 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kckp\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707574 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707644 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707668 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707690 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.707859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.707720 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.721065 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.721036 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0b98c1-openshift-default-dockercfg-pm5jk\"" Nov 03 02:25:03.763934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.763893 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:25:03.808636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808591 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808661 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808678 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808698 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808719 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808771 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808798 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.808859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.808834 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.809743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.809710 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.810363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.810337 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kckp\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.811698 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.811673 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.811963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.811939 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.812206 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.812186 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.812434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.812414 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.830581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.816457 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.830581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.822908 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.891133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.890773 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kckp\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.894211 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.894179 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\") pod \"e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:03.927063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.925320 4992 patch_prober.go:29] interesting pod/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:25:03.927063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.925398 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:25:03.969765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:03.969729 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:04.406580 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:04.406475 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:25:04.439568 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:25:04.439523 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2df1229_3e31_4459_9e07_bd27af0e9581.slice/crio-f78fe9d9c141e5bc1f0980763c01dc7e3482a17001b5cb179d508f83737c7770 WatchSource:0}: Error finding container f78fe9d9c141e5bc1f0980763c01dc7e3482a17001b5cb179d508f83737c7770: Status 404 returned error can't find the container with id f78fe9d9c141e5bc1f0980763c01dc7e3482a17001b5cb179d508f83737c7770 Nov 03 02:25:04.442301 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:04.442259 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:04.442401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:04.442335 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:04.442401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:04.442369 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:04.493287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:04.493253 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerStarted","Data":"f78fe9d9c141e5bc1f0980763c01dc7e3482a17001b5cb179d508f83737c7770"} Nov 03 02:25:05.500927 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:05.500884 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerStarted","Data":"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7"} Nov 03 02:25:05.626579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:05.626493 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podStartSLOduration=2.626477111 podStartE2EDuration="2.626477111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:05.624472756 +0000 UTC m=+4041.203295343" watchObservedRunningTime="2026-11-03 02:25:05.626477111 +0000 UTC m=+4041.205299697" Nov 03 02:25:05.969970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:05.969927 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:05.976988 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:05.976950 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:06.507827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:06.507793 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:06.508888 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:06.508864 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:25:07.515816 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.515684 4992 generic.go:356] "Generic (PLEG): container finished" podID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerID="a42cd4590b44dc64e8936f62668ea94f6d76a7072872c72b3793faf1f059173e" exitCode=0 Nov 03 02:25:07.515816 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.515751 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerDied","Data":"a42cd4590b44dc64e8936f62668ea94f6d76a7072872c72b3793faf1f059173e"} Nov 03 02:25:07.516328 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.515841 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" event={"ID":"b50dbdcc-3f3e-4f33-bdb2-c5091025212c","Type":"ContainerDied","Data":"c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e"} Nov 03 02:25:07.516328 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.515861 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9f74872f59c2668013199028b6bb3018017388f47403894a2f7e9ef9a58008e" Nov 03 02:25:07.516328 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.515875 4992 scope.go:122] "RemoveContainer" containerID="d4f067c86a8bddecf591ef599672c7cdae7499d322c00ac2bcdf89fa8105ffaa" Nov 03 02:25:07.539394 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.539369 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:25:07.563851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.563811 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.563977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.563887 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.563977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.563928 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564087 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564021 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564087 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564066 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564091 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564129 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564156 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564220 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:07.564358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564238 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\") pod \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\" (UID: \"b50dbdcc-3f3e-4f33-bdb2-c5091025212c\") " Nov 03 02:25:07.564358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564273 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:07.564358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564314 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:07.564594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564463 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:07.564594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564529 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:25:07.565195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564963 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.565195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.564987 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.565195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.565003 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.565195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.565015 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.565195 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.565027 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.566573 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.566536 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:25:07.566695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.566603 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:25:07.566880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.566861 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:07.566942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.566910 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79" pod "b50dbdcc-3f3e-4f33-bdb2-c5091025212c" (UID: "b50dbdcc-3f3e-4f33-bdb2-c5091025212c"). InnerVolumeSpecName "kube-api-access-brz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:25:07.666746 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.666701 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.666746 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.666735 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.666746 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.666746 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brz79\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-kube-api-access-brz79\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:07.666746 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:07.666755 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50dbdcc-3f3e-4f33-bdb2-c5091025212c-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:25:08.524098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:08.524064 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb" Nov 03 02:25:08.821234 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:08.821138 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:25:08.826992 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:08.826946 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3fb07e-openshift-default-86f675b965-55zpb"] Nov 03 02:25:10.160866 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.160829 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:25:10.161651 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161627 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.161651 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161651 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.161840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161674 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.161840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161682 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.161946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161888 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.161946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.161903 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" containerName="istio-proxy" Nov 03 02:25:10.166776 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.166749 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.220383 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.220349 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:25:10.268080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.268042 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d6fbd7-serving-cert\"" Nov 03 02:25:10.300089 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.300046 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.300261 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.300104 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.300261 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.300216 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxdt\" (UniqueName: \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.300386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.300342 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.401394 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.401351 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.401636 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:10.401540 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-serving-cert: secret "maas-api-e2e-inf-a-d6fbd7-serving-cert" not found Nov 03 02:25:10.401636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.401550 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.401636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.401597 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.401827 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:10.401650 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls podName:5126e773-ed2a-4f73-9d2b-c5fbd605803f nodeName:}" failed. No retries permitted until 2026-11-03 02:25:10.901625905 +0000 UTC m=+4046.480448489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls") pod "maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" (UID: "5126e773-ed2a-4f73-9d2b-c5fbd605803f") : secret "maas-api-e2e-inf-a-d6fbd7-serving-cert" not found Nov 03 02:25:10.401827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.401673 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxdt\" (UniqueName: \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.402401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.402371 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.405156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.405132 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.467796 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.467767 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxdt\" (UniqueName: \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.595486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.595449 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50dbdcc-3f3e-4f33-bdb2-c5091025212c" path="/var/lib/kubelet/pods/b50dbdcc-3f3e-4f33-bdb2-c5091025212c/volumes" Nov 03 02:25:10.909262 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.909142 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:10.911877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:10.911851 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:11.081137 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:11.081103 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:11.458434 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:25:11.458399 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5126e773_ed2a_4f73_9d2b_c5fbd605803f.slice/crio-d4a5cf524fdd50ee040f11e838c558528ff8742f1dd91951bd82bab1db351cbd WatchSource:0}: Error finding container d4a5cf524fdd50ee040f11e838c558528ff8742f1dd91951bd82bab1db351cbd: Status 404 returned error can't find the container with id d4a5cf524fdd50ee040f11e838c558528ff8742f1dd91951bd82bab1db351cbd Nov 03 02:25:11.460146 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:11.460123 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:25:11.461603 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:11.461581 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:25:11.550665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:11.550631 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" event={"ID":"5126e773-ed2a-4f73-9d2b-c5fbd605803f","Type":"ContainerStarted","Data":"d4a5cf524fdd50ee040f11e838c558528ff8742f1dd91951bd82bab1db351cbd"} Nov 03 02:25:12.559475 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:12.559434 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" event={"ID":"5126e773-ed2a-4f73-9d2b-c5fbd605803f","Type":"ContainerStarted","Data":"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a"} Nov 03 02:25:12.559898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:12.559512 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:12.709719 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:12.709655 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" podStartSLOduration=1.773301133 podStartE2EDuration="2.709642193s" totalImagesPullingTime="936.34106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:12.709205056 +0000 UTC m=+4048.288027642" watchObservedRunningTime="2026-11-03 02:25:12.709642193 +0000 UTC m=+4048.288464779" Nov 03 02:25:18.610029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:18.609998 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:25:32.152340 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:32.152298 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:25:34.443629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.443586 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:25:34.448461 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.448434 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.503174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.503127 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:25:34.542167 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.542126 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jksr\" (UniqueName: \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.542385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.542325 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.542385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.542371 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.542509 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.542485 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.550459 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.550420 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c56865-serving-cert\"" Nov 03 02:25:34.643471 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.643421 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jksr\" (UniqueName: \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.643726 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.643608 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.643726 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.643670 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.643726 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.643718 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.644365 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:34.644063 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-serving-cert: secret "maas-api-e2e-inf-a-c56865-serving-cert" not found Nov 03 02:25:34.644365 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:25:34.644147 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls podName:c6600d9d-51ed-4044-9695-12ace0eedf1a nodeName:}" failed. No retries permitted until 2026-11-03 02:25:35.144124982 +0000 UTC m=+4070.722947565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls") pod "maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" (UID: "c6600d9d-51ed-4044-9695-12ace0eedf1a") : secret "maas-api-e2e-inf-a-c56865-serving-cert" not found Nov 03 02:25:34.644603 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.644517 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.647335 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.647302 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.710658 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.710541 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jksr\" (UniqueName: \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:34.757266 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.757228 4992 generic.go:356] "Generic (PLEG): container finished" podID="2f242d8f-e746-48ef-a383-9e06f3565481" containerID="5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218" exitCode=0 Nov 03 02:25:34.757661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.757638 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerDied","Data":"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218"} Nov 03 02:25:34.758285 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.758260 4992 scope.go:122] "RemoveContainer" containerID="5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218" Nov 03 02:25:34.764829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.764783 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:34.765669 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.765639 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:34.765858 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:34.765838 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:25:35.153899 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:35.153795 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:35.160936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:35.160872 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") pod \"maas-api-e2e-inf-a-c56865-6545584cd9-5frtb\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:35.364999 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:35.363613 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:35.759784 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:25:35.759736 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6600d9d_51ed_4044_9695_12ace0eedf1a.slice/crio-3cdb23c4cfc088857c96204210e184dcb76d69a50b4d9a6add535eb31515b50e WatchSource:0}: Error finding container 3cdb23c4cfc088857c96204210e184dcb76d69a50b4d9a6add535eb31515b50e: Status 404 returned error can't find the container with id 3cdb23c4cfc088857c96204210e184dcb76d69a50b4d9a6add535eb31515b50e Nov 03 02:25:35.762913 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:35.762667 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:25:35.767863 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:35.767831 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerStarted","Data":"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30"} Nov 03 02:25:36.617491 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.617454 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:25:36.619262 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.619228 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:25:36.619406 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.619290 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 02:25:36.791822 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.791767 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" event={"ID":"c6600d9d-51ed-4044-9695-12ace0eedf1a","Type":"ContainerStarted","Data":"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c"} Nov 03 02:25:36.792314 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.791839 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" event={"ID":"c6600d9d-51ed-4044-9695-12ace0eedf1a","Type":"ContainerStarted","Data":"3cdb23c4cfc088857c96204210e184dcb76d69a50b4d9a6add535eb31515b50e"} Nov 03 02:25:36.792314 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.791891 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:25:36.915000 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:36.914950 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" podStartSLOduration=2.0671261 podStartE2EDuration="2.914935658s" totalImagesPullingTime="847.809558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:36.91270126 +0000 UTC m=+4072.491523847" watchObservedRunningTime="2026-11-03 02:25:36.914935658 +0000 UTC m=+4072.493758244" Nov 03 02:25:37.618706 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:37.618613 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:25:37.618706 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:37.618674 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 02:25:38.617864 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:38.617827 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:25:38.618375 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:38.617887 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 02:25:38.895890 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:38.895804 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:25:39.618615 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:39.618540 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:25:39.619117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:39.618630 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 02:25:40.618232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:40.618188 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 02:25:40.618425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:40.618255 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 02:25:41.668520 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:41.668457 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:25:41.670367 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:41.670343 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:25:42.804249 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:25:42.804219 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:26:02.157845 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:26:02.157759 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:26:32.151314 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:26:32.151265 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:27:02.151860 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:27:02.151818 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:27:06.389822 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.389788 4992 generic.go:356] "Generic (PLEG): container finished" podID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerID="2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7" exitCode=0 Nov 03 02:27:06.390283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.389846 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerDied","Data":"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7"} Nov 03 02:27:06.390283 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.390274 4992 scope.go:122] "RemoveContainer" containerID="2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7" Nov 03 02:27:06.391991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.391958 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:06.392060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.392033 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:06.392098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.392069 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:27:06.508001 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:06.507959 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:27:07.397695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:07.397651 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerStarted","Data":"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b"} Nov 03 02:27:07.985374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:07.985342 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:27:07.986047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:07.986003 4992 patch_prober.go:29] interesting pod/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 02:27:07.986220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:07.986077 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 02:27:08.980045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:08.980003 4992 patch_prober.go:29] interesting pod/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 02:27:08.980830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:08.980791 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 02:27:09.978586 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:09.971992 4992 patch_prober.go:29] interesting pod/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 02:27:09.978586 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:09.972061 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 02:27:10.987756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:10.987715 4992 patch_prober.go:29] interesting pod/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 02:27:10.988267 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:10.987856 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 02:27:11.979407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:11.978946 4992 patch_prober.go:29] interesting pod/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 02:27:11.979407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:11.979008 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 02:27:11.987585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:11.987398 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:27:11.991917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:11.991892 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.038775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.038704 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:27:12.058231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.058186 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.058403 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.058240 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.058403 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.058343 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vdj\" (UniqueName: \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.058403 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.058377 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.086946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.086889 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b6c56b-serving-cert\"" Nov 03 02:27:12.159934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.159886 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.159934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.159937 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.160198 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.159981 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vdj\" (UniqueName: \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.160198 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.160008 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.160198 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:27:12.160033 4992 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-serving-cert: secret "maas-api-e2e-extmod-b-b6c56b-serving-cert" not found Nov 03 02:27:12.160198 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:27:12.160119 4992 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls podName:0e8030fa-2d3c-4e8f-a2d4-b47eed643e35 nodeName:}" failed. No retries permitted until 2026-11-03 02:27:12.660095657 +0000 UTC m=+4168.238918223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls") pod "maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" (UID: "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35") : secret "maas-api-e2e-extmod-b-b6c56b-serving-cert" not found Nov 03 02:27:12.160719 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.160688 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.193095 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.193061 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.226737 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.226697 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vdj\" (UniqueName: \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.680856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.680807 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.683679 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.683642 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.905420 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.905386 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:12.979606 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.979573 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:27:12.980757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.980732 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:27:12.981487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:12.981446 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:27:13.285354 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:27:13.285322 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8030fa_2d3c_4e8f_a2d4_b47eed643e35.slice/crio-d9f8c5f6c76e6c4e7fb4bdccd6f61a4f4dffbcc2893bfbe301d466c9e94c83dc WatchSource:0}: Error finding container d9f8c5f6c76e6c4e7fb4bdccd6f61a4f4dffbcc2893bfbe301d466c9e94c83dc: Status 404 returned error can't find the container with id d9f8c5f6c76e6c4e7fb4bdccd6f61a4f4dffbcc2893bfbe301d466c9e94c83dc Nov 03 02:27:13.286751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:13.286718 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:27:13.438131 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:13.438088 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" event={"ID":"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35","Type":"ContainerStarted","Data":"d9f8c5f6c76e6c4e7fb4bdccd6f61a4f4dffbcc2893bfbe301d466c9e94c83dc"} Nov 03 02:27:15.452252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:15.452210 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" event={"ID":"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35","Type":"ContainerStarted","Data":"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057"} Nov 03 02:27:15.452758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:15.452323 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:15.575966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:15.575907 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" podStartSLOduration=3.457574471 podStartE2EDuration="4.575892299s" totalImagesPullingTime="1.118317828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:27:15.573341891 +0000 UTC m=+4171.152164478" watchObservedRunningTime="2026-11-03 02:27:15.575892299 +0000 UTC m=+4171.154714885" Nov 03 02:27:21.466729 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:21.466697 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:27:32.151664 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:27:32.151625 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:27:45.216758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:45.216603 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:27:45.219971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:45.219944 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:27:45.778653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:45.778620 4992 scope.go:122] "RemoveContainer" containerID="a42cd4590b44dc64e8936f62668ea94f6d76a7072872c72b3793faf1f059173e" Nov 03 02:27:45.788840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:45.788805 4992 scope.go:122] "RemoveContainer" containerID="46e7b03d7ec868f188b0858bb82f07b1abc78252fb19e87147041ca1f17c24b1" Nov 03 02:27:45.798037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:27:45.798008 4992 scope.go:122] "RemoveContainer" containerID="9d71d70fb0e9a9921b4b447378a4b4c00bd50fb7edb00fc0c25e66bedcb8205a" Nov 03 02:28:02.151275 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:28:02.151235 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:28:21.859511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:21.859471 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:28:21.860067 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:21.859729 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" containerID="cri-o://e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b" gracePeriod=30 Nov 03 02:28:27.141638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.141575 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:28:27.336212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336164 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.336398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336273 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.336398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336390 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.336515 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336425 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.336515 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336434 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:27.336655 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336594 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.337125 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.336815 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.337125 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337088 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:27.337302 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337203 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.337540 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337513 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:27.337640 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337548 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:27.337640 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337573 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.337761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337644 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\") pod \"c2df1229-3e31-4459-9e07-bd27af0e9581\" (UID: \"c2df1229-3e31-4459-9e07-bd27af0e9581\") " Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.337904 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.338285 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2df1229-3e31-4459-9e07-bd27af0e9581-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.338312 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.338328 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.338343 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.338385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.338357 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.339435 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.339413 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "kube-api-access-8kckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:27.339760 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.339732 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:27.340191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.340167 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:28:27.340876 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.340849 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy" pod "c2df1229-3e31-4459-9e07-bd27af0e9581" (UID: "c2df1229-3e31-4459-9e07-bd27af0e9581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:28:27.440097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.440052 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kckp\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-kube-api-access-8kckp\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.440097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.440091 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.440097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.440104 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.440428 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.440116 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2df1229-3e31-4459-9e07-bd27af0e9581-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:27.903972 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.903877 4992 generic.go:356] "Generic (PLEG): container finished" podID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerID="e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b" exitCode=0 Nov 03 02:28:27.903972 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.903928 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" Nov 03 02:28:27.904191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.903965 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerDied","Data":"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b"} Nov 03 02:28:27.904191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.904023 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz" event={"ID":"c2df1229-3e31-4459-9e07-bd27af0e9581","Type":"ContainerDied","Data":"f78fe9d9c141e5bc1f0980763c01dc7e3482a17001b5cb179d508f83737c7770"} Nov 03 02:28:27.904191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.904050 4992 scope.go:122] "RemoveContainer" containerID="e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b" Nov 03 02:28:27.916004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.915982 4992 scope.go:122] "RemoveContainer" containerID="2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7" Nov 03 02:28:27.927888 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.927866 4992 scope.go:122] "RemoveContainer" containerID="e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b" Nov 03 02:28:27.928217 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:28:27.928193 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b\": container with ID starting with e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b not found: ID does not exist" containerID="e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b" Nov 03 02:28:27.928331 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.928231 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b"} err="failed to get container status \"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b\": rpc error: code = NotFound desc = could not find container \"e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b\": container with ID starting with e347926c5c1d613add213cf876f4aff5a0eca0ab93a1f5df0bdebc6b48c9103b not found: ID does not exist" Nov 03 02:28:27.928331 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.928256 4992 scope.go:122] "RemoveContainer" containerID="2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7" Nov 03 02:28:27.928625 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:28:27.928605 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7\": container with ID starting with 2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7 not found: ID does not exist" containerID="2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7" Nov 03 02:28:27.928697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:27.928633 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7"} err="failed to get container status \"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7\": rpc error: code = NotFound desc = could not find container \"2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7\": container with ID starting with 2e12d4130f03dcb4b5c4dec04a320ce1f12af2f9cf805eec9f90f8212e6c5fb7 not found: ID does not exist" Nov 03 02:28:28.127628 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:28.127584 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:28:28.179420 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:28.179382 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0b98c1-openshift-default-567b98f789-jcrbz"] Nov 03 02:28:28.595766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:28.595686 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" path="/var/lib/kubelet/pods/c2df1229-3e31-4459-9e07-bd27af0e9581/volumes" Nov 03 02:28:32.152473 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:28:32.152429 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:28:39.431764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.431722 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:28:39.432300 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.431965 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" podUID="c6600d9d-51ed-4044-9695-12ace0eedf1a" containerName="maas-api" containerID="cri-o://7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c" gracePeriod=30 Nov 03 02:28:39.771398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.771365 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:28:39.900469 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.900435 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\") pod \"c6600d9d-51ed-4044-9695-12ace0eedf1a\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " Nov 03 02:28:39.900930 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.900897 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\" (UniqueName: \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\") pod \"c6600d9d-51ed-4044-9695-12ace0eedf1a\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " Nov 03 02:28:39.901053 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.900945 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\") pod \"c6600d9d-51ed-4044-9695-12ace0eedf1a\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " Nov 03 02:28:39.901053 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.901044 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") pod \"c6600d9d-51ed-4044-9695-12ace0eedf1a\" (UID: \"c6600d9d-51ed-4044-9695-12ace0eedf1a\") " Nov 03 02:28:39.901532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.901503 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca" pod "c6600d9d-51ed-4044-9695-12ace0eedf1a" (UID: "c6600d9d-51ed-4044-9695-12ace0eedf1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:28:39.903533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.903365 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs" pod "c6600d9d-51ed-4044-9695-12ace0eedf1a" (UID: "c6600d9d-51ed-4044-9695-12ace0eedf1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:28:39.904271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.904247 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls" pod "c6600d9d-51ed-4044-9695-12ace0eedf1a" (UID: "c6600d9d-51ed-4044-9695-12ace0eedf1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:28:39.905833 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.905792 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr" pod "c6600d9d-51ed-4044-9695-12ace0eedf1a" (UID: "c6600d9d-51ed-4044-9695-12ace0eedf1a"). InnerVolumeSpecName "kube-api-access-4jksr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:28:39.995732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.995639 4992 generic.go:356] "Generic (PLEG): container finished" podID="c6600d9d-51ed-4044-9695-12ace0eedf1a" containerID="7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c" exitCode=0 Nov 03 02:28:39.995732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.995693 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" Nov 03 02:28:39.995732 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.995699 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" event={"ID":"c6600d9d-51ed-4044-9695-12ace0eedf1a","Type":"ContainerDied","Data":"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c"} Nov 03 02:28:39.996007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.995760 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb" event={"ID":"c6600d9d-51ed-4044-9695-12ace0eedf1a","Type":"ContainerDied","Data":"3cdb23c4cfc088857c96204210e184dcb76d69a50b4d9a6add535eb31515b50e"} Nov 03 02:28:39.996007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:39.995784 4992 scope.go:122] "RemoveContainer" containerID="7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c" Nov 03 02:28:40.003203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.003163 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.003203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.003198 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jksr\" (UniqueName: \"kubernetes.io/projected/c6600d9d-51ed-4044-9695-12ace0eedf1a-kube-api-access-4jksr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.003440 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.003217 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6600d9d-51ed-4044-9695-12ace0eedf1a-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.003440 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.003231 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6600d9d-51ed-4044-9695-12ace0eedf1a-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:28:40.007231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.007211 4992 scope.go:122] "RemoveContainer" containerID="7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c" Nov 03 02:28:40.007595 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:28:40.007566 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c\": container with ID starting with 7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c not found: ID does not exist" containerID="7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c" Nov 03 02:28:40.007661 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.007608 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c"} err="failed to get container status \"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c\": rpc error: code = NotFound desc = could not find container \"7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c\": container with ID starting with 7f4a66cbcab47679d0830d4ad6f2b084321c86ae7d3b348a8652bcb8dc09e45c not found: ID does not exist" Nov 03 02:28:40.172410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.172371 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:28:40.176824 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.176795 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c56865-6545584cd9-5frtb"] Nov 03 02:28:40.595546 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:28:40.595510 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6600d9d-51ed-4044-9695-12ace0eedf1a" path="/var/lib/kubelet/pods/c6600d9d-51ed-4044-9695-12ace0eedf1a/volumes" Nov 03 02:29:02.153095 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:29:02.153006 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:29:32.151844 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:29:32.151800 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:29:37.563433 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.563391 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:29:37.564028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564011 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564028 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564039 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564044 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564062 4992 cpu_manager.go:414] "containerMap: removing container" podUID="c6600d9d-51ed-4044-9695-12ace0eedf1a" containerName="maas-api" Nov 03 02:29:37.564122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564067 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6600d9d-51ed-4044-9695-12ace0eedf1a" containerName="maas-api" Nov 03 02:29:37.564260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564150 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564159 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2df1229-3e31-4459-9e07-bd27af0e9581" containerName="istio-proxy" Nov 03 02:29:37.564260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.564168 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6600d9d-51ed-4044-9695-12ace0eedf1a" containerName="maas-api" Nov 03 02:29:37.567586 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.567550 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609279 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609345 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtt8x\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609423 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609454 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609489 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609806 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609637 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609806 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609709 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609842 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.609910 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.609892 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.678838 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.678795 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:29:37.710947 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.710907 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.710976 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtt8x\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711001 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711029 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711067 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711106 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711149 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711246 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711301 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711474 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711366 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711679 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711581 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711679 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711599 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711798 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711735 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.711861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.711841 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.713926 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.713905 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.714250 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.714229 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.774698 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.774662 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.774848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.774750 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtt8x\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bccqr\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:37.881668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:37.881539 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:38.309445 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:29:38.309408 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ba8a9d_c4e3_4879_8cdd_f0cf4a6fdb26.slice/crio-db572fd39bedfc98126f1ad84b4e800bb17b747a986256298591611f06de8782 WatchSource:0}: Error finding container db572fd39bedfc98126f1ad84b4e800bb17b747a986256298591611f06de8782: Status 404 returned error can't find the container with id db572fd39bedfc98126f1ad84b4e800bb17b747a986256298591611f06de8782 Nov 03 02:29:38.312678 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.312640 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:38.312766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.312714 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:38.312766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.312746 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:29:38.312834 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.312783 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:29:38.421869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.421813 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" event={"ID":"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26","Type":"ContainerStarted","Data":"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286"} Nov 03 02:29:38.422078 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.421897 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" event={"ID":"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26","Type":"ContainerStarted","Data":"db572fd39bedfc98126f1ad84b4e800bb17b747a986256298591611f06de8782"} Nov 03 02:29:38.546987 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.546932 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podStartSLOduration=1.546917949 podStartE2EDuration="1.546917949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:38.54610021 +0000 UTC m=+4314.124922802" watchObservedRunningTime="2026-11-03 02:29:38.546917949 +0000 UTC m=+4314.125740537" Nov 03 02:29:38.882475 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:38.882435 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:39.888817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:39.888772 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:39.889297 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:39.888850 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:40.886690 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:40.882993 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:40.886690 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:40.883064 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:41.884007 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:41.883964 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:41.884658 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:41.884621 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:42.882072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:42.882031 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:42.882348 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:42.882319 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:43.883584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:43.882948 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:43.883584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:43.883013 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:44.882881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:44.882842 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:44.883067 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:44.882896 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:45.881938 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:45.881901 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bccqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 02:29:45.882353 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:45.881964 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 02:29:47.473447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:47.473365 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:47.474015 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:47.473899 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:47.475067 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:47.475039 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:29:48.865414 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:48.865363 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:29:48.866846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:48.866526 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" podUID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" containerName="maas-api" containerID="cri-o://64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a" gracePeriod=30 Nov 03 02:29:49.205057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.205019 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:29:49.296517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.296484 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") pod \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " Nov 03 02:29:49.296730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.296547 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\") pod \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " Nov 03 02:29:49.296730 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.296660 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\") pod \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " Nov 03 02:29:49.296932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.296778 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\" (UniqueName: \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\") pod \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\" (UID: \"5126e773-ed2a-4f73-9d2b-c5fbd605803f\") " Nov 03 02:29:49.296932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.296897 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca" pod "5126e773-ed2a-4f73-9d2b-c5fbd605803f" (UID: "5126e773-ed2a-4f73-9d2b-c5fbd605803f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:29:49.297337 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.297309 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5126e773-ed2a-4f73-9d2b-c5fbd605803f-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:49.299156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.299120 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt" pod "5126e773-ed2a-4f73-9d2b-c5fbd605803f" (UID: "5126e773-ed2a-4f73-9d2b-c5fbd605803f"). InnerVolumeSpecName "kube-api-access-6jxdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:29:49.299390 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.299366 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls" pod "5126e773-ed2a-4f73-9d2b-c5fbd605803f" (UID: "5126e773-ed2a-4f73-9d2b-c5fbd605803f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:49.299504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.299387 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs" pod "5126e773-ed2a-4f73-9d2b-c5fbd605803f" (UID: "5126e773-ed2a-4f73-9d2b-c5fbd605803f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:29:49.398752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.398709 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:49.398752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.398751 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxdt\" (UniqueName: \"kubernetes.io/projected/5126e773-ed2a-4f73-9d2b-c5fbd605803f-kube-api-access-6jxdt\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:49.398965 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.398767 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5126e773-ed2a-4f73-9d2b-c5fbd605803f-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:29:49.495230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.495117 4992 generic.go:356] "Generic (PLEG): container finished" podID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" containerID="64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a" exitCode=0 Nov 03 02:29:49.495230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.495175 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" Nov 03 02:29:49.495230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.495207 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" event={"ID":"5126e773-ed2a-4f73-9d2b-c5fbd605803f","Type":"ContainerDied","Data":"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a"} Nov 03 02:29:49.495468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.495274 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8" event={"ID":"5126e773-ed2a-4f73-9d2b-c5fbd605803f","Type":"ContainerDied","Data":"d4a5cf524fdd50ee040f11e838c558528ff8742f1dd91951bd82bab1db351cbd"} Nov 03 02:29:49.495468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.495301 4992 scope.go:122] "RemoveContainer" containerID="64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a" Nov 03 02:29:49.506761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.506741 4992 scope.go:122] "RemoveContainer" containerID="64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a" Nov 03 02:29:49.507068 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:29:49.507029 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a\": container with ID starting with 64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a not found: ID does not exist" containerID="64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a" Nov 03 02:29:49.507172 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.507070 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a"} err="failed to get container status \"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a\": rpc error: code = NotFound desc = could not find container \"64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a\": container with ID starting with 64b5c446828a7d5c0b82ab29a5fbb8610e304c95defbb621954f80262655064a not found: ID does not exist" Nov 03 02:29:49.666688 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.666651 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:29:49.670751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:49.670722 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6fbd7-f5b955f7-h87l8"] Nov 03 02:29:50.598088 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:29:50.598038 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" path="/var/lib/kubelet/pods/5126e773-ed2a-4f73-9d2b-c5fbd605803f/volumes" Nov 03 02:30:00.263990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.263948 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:00.264533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.264519 4992 cpu_manager.go:414] "containerMap: removing container" podUID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" containerName="maas-api" Nov 03 02:30:00.264619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.264534 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" containerName="maas-api" Nov 03 02:30:00.264663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.264636 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="5126e773-ed2a-4f73-9d2b-c5fbd605803f" containerName="maas-api" Nov 03 02:30:00.268203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.268179 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.278760 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.278627 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:00.283281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.283258 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:00.283400 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.283378 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.287114 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.287092 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.315450 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.315414 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:00.320090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.320061 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.327180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.327152 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:00.332458 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.332424 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:00.336023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.335757 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9dd\" (UniqueName: \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.336023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.335881 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.338140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.337544 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbc7d\" (UniqueName: \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.338140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.337624 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.338140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.337676 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.338140 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.337826 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt5j\" (UniqueName: \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.351549 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.351514 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.364093 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.363108 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:30:00.369942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.369903 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:00.381741 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.381715 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:00.381907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.381760 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:00.381907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.381847 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.383655 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.383629 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:00.387113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.387082 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:00.411339 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.411303 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:00.439637 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439598 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbt2\" (UniqueName: \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.439830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439665 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.439830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439716 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt5j\" (UniqueName: \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.439830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439781 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9dd\" (UniqueName: \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.439830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439817 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.440160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.439914 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.440160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440030 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.440160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440074 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5prr\" (UniqueName: \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.440160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440113 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbc7d\" (UniqueName: \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.441793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440552 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwhj\" (UniqueName: \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.441793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440621 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.441793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.440687 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.441793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.441147 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.441793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.441730 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.442361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.442287 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.503605 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.503133 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt5j\" (UniqueName: \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.503605 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.503233 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9dd\" (UniqueName: \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\") pod \"maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.503605 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.503263 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbc7d\" (UniqueName: \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\") pod \"maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.542173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542066 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5prr\" (UniqueName: \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.542173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542140 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwhj\" (UniqueName: \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.542501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542248 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbt2\" (UniqueName: \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.542501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542285 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.542501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542412 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.542501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.542480 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.543211 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.543183 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.543359 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.543337 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.543408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.543355 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.587374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.587332 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:00.597979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.597952 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:00.604772 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.604737 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:00.606191 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.606160 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwhj\" (UniqueName: \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.606753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.606729 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5prr\" (UniqueName: \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.612051 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.610371 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbt2\" (UniqueName: \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.648019 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.647762 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:00.679943 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.679718 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:00.701392 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:00.699461 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:00.994603 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:30:00.991315 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2665b1e_0135_4498_b431_768a205a6185.slice/crio-d997a4823c5c362e36212e427dbd3b94d20bceb07d68acf423b5f945bf8754e6 WatchSource:0}: Error finding container d997a4823c5c362e36212e427dbd3b94d20bceb07d68acf423b5f945bf8754e6: Status 404 returned error can't find the container with id d997a4823c5c362e36212e427dbd3b94d20bceb07d68acf423b5f945bf8754e6 Nov 03 02:30:01.011784 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.011685 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:01.025219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.025187 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:01.028637 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:30:01.028606 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946c4ca4_b5cb_495c_ae44_391634a6d663.slice/crio-487c80cefb40332fc8acb8cd048ca81ba1c4e946f9d42928ef74bc436fa31b79 WatchSource:0}: Error finding container 487c80cefb40332fc8acb8cd048ca81ba1c4e946f9d42928ef74bc436fa31b79: Status 404 returned error can't find the container with id 487c80cefb40332fc8acb8cd048ca81ba1c4e946f9d42928ef74bc436fa31b79 Nov 03 02:30:01.028790 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.028765 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:01.084011 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.083968 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:01.084785 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:30:01.084744 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde70cdc4_127f_40e4_a3f6_7a65404852a2.slice/crio-bb5d52edd43cc05afdfb0a6837c1d01dbccb8687eda840d761a4b97c5b109393 WatchSource:0}: Error finding container bb5d52edd43cc05afdfb0a6837c1d01dbccb8687eda840d761a4b97c5b109393: Status 404 returned error can't find the container with id bb5d52edd43cc05afdfb0a6837c1d01dbccb8687eda840d761a4b97c5b109393 Nov 03 02:30:01.118835 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:30:01.118801 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243c74d3_afa4_4fac_96e7_f57aa76d669c.slice/crio-adcdac0522418de6756c0196e5a2e3972aad7b19708f53c9d6b436cccd8161c2 WatchSource:0}: Error finding container adcdac0522418de6756c0196e5a2e3972aad7b19708f53c9d6b436cccd8161c2: Status 404 returned error can't find the container with id adcdac0522418de6756c0196e5a2e3972aad7b19708f53c9d6b436cccd8161c2 Nov 03 02:30:01.121201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.121177 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:01.124968 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:30:01.124927 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebd92a5_ae5d_4a4e_8e62_0d0beeba8d43.slice/crio-6be950e49b822a726bb7838aa678fb48fae076d62c2f3a3448684bb3373e8ab3 WatchSource:0}: Error finding container 6be950e49b822a726bb7838aa678fb48fae076d62c2f3a3448684bb3373e8ab3: Status 404 returned error can't find the container with id 6be950e49b822a726bb7838aa678fb48fae076d62c2f3a3448684bb3373e8ab3 Nov 03 02:30:01.128829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.128803 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:01.575575 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.575529 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerStarted","Data":"bb5d52edd43cc05afdfb0a6837c1d01dbccb8687eda840d761a4b97c5b109393"} Nov 03 02:30:01.577550 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.577522 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerStarted","Data":"487c80cefb40332fc8acb8cd048ca81ba1c4e946f9d42928ef74bc436fa31b79"} Nov 03 02:30:01.578992 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.578967 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerStarted","Data":"6be950e49b822a726bb7838aa678fb48fae076d62c2f3a3448684bb3373e8ab3"} Nov 03 02:30:01.581394 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.581369 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerStarted","Data":"adcdac0522418de6756c0196e5a2e3972aad7b19708f53c9d6b436cccd8161c2"} Nov 03 02:30:01.583306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.583276 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerStarted","Data":"b43f9b2390e7dcbb761d3bbe13fc12d076dcc6886d3b3fa96c213c4dd4dbaac7"} Nov 03 02:30:01.583415 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.583326 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerStarted","Data":"1e19d65c2f859e04bc09cb3baad22e0a8deb18499bb216e80c6d542a9a4abc82"} Nov 03 02:30:01.584913 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.584891 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" event={"ID":"e2665b1e-0135-4498-b431-768a205a6185","Type":"ContainerStarted","Data":"d997a4823c5c362e36212e427dbd3b94d20bceb07d68acf423b5f945bf8754e6"} Nov 03 02:30:01.701968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:01.701918 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" podStartSLOduration=1.701904203 podStartE2EDuration="1.701904203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:01.700774078 +0000 UTC m=+4337.279596665" watchObservedRunningTime="2026-11-03 02:30:01.701904203 +0000 UTC m=+4337.280726789" Nov 03 02:30:02.172669 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:02.171276 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:30:02.595745 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.595658 4992 generic.go:356] "Generic (PLEG): container finished" podID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerID="f6ac49bf49c2a530727768dbad5bc998312ac4a7bd39852739a447d5805e02b6" exitCode=60 Nov 03 02:30:02.596174 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.596044 4992 scope.go:122] "RemoveContainer" containerID="f6ac49bf49c2a530727768dbad5bc998312ac4a7bd39852739a447d5805e02b6" Nov 03 02:30:02.596290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.596262 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerDied","Data":"f6ac49bf49c2a530727768dbad5bc998312ac4a7bd39852739a447d5805e02b6"} Nov 03 02:30:02.597437 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.597409 4992 generic.go:356] "Generic (PLEG): container finished" podID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerID="2a7eea51031ae286d1e7b5c2c7cb41457445c9bee865ddb8fe32bc904c8ab1c8" exitCode=60 Nov 03 02:30:02.597592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.597485 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerDied","Data":"2a7eea51031ae286d1e7b5c2c7cb41457445c9bee865ddb8fe32bc904c8ab1c8"} Nov 03 02:30:02.597765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.597702 4992 scope.go:122] "RemoveContainer" containerID="2a7eea51031ae286d1e7b5c2c7cb41457445c9bee865ddb8fe32bc904c8ab1c8" Nov 03 02:30:02.599293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.599212 4992 generic.go:356] "Generic (PLEG): container finished" podID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerID="c28f48c089789cd3bdbda9a14ac01371ec01ffee23abbfc165a40d7658ede386" exitCode=60 Nov 03 02:30:02.599293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.599251 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerDied","Data":"c28f48c089789cd3bdbda9a14ac01371ec01ffee23abbfc165a40d7658ede386"} Nov 03 02:30:02.599505 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.599490 4992 scope.go:122] "RemoveContainer" containerID="c28f48c089789cd3bdbda9a14ac01371ec01ffee23abbfc165a40d7658ede386" Nov 03 02:30:02.601038 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.601014 4992 generic.go:356] "Generic (PLEG): container finished" podID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerID="b43f9b2390e7dcbb761d3bbe13fc12d076dcc6886d3b3fa96c213c4dd4dbaac7" exitCode=60 Nov 03 02:30:02.601143 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.601088 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerDied","Data":"b43f9b2390e7dcbb761d3bbe13fc12d076dcc6886d3b3fa96c213c4dd4dbaac7"} Nov 03 02:30:02.601300 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.601279 4992 scope.go:122] "RemoveContainer" containerID="b43f9b2390e7dcbb761d3bbe13fc12d076dcc6886d3b3fa96c213c4dd4dbaac7" Nov 03 02:30:02.602655 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.602635 4992 generic.go:356] "Generic (PLEG): container finished" podID="e2665b1e-0135-4498-b431-768a205a6185" containerID="76e0e3aed0b07b1bbce654f1daf35e055e799e81f3498a2624ba486089e94bc8" exitCode=60 Nov 03 02:30:02.602758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.602684 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" event={"ID":"e2665b1e-0135-4498-b431-768a205a6185","Type":"ContainerDied","Data":"76e0e3aed0b07b1bbce654f1daf35e055e799e81f3498a2624ba486089e94bc8"} Nov 03 02:30:02.602863 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.602847 4992 scope.go:122] "RemoveContainer" containerID="76e0e3aed0b07b1bbce654f1daf35e055e799e81f3498a2624ba486089e94bc8" Nov 03 02:30:02.604288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.604270 4992 generic.go:356] "Generic (PLEG): container finished" podID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerID="c3190fb03f66d71d20bf49c25af06e84b4f5019e8c8c9e39405f722d1a4706e9" exitCode=60 Nov 03 02:30:02.604377 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.604310 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerDied","Data":"c3190fb03f66d71d20bf49c25af06e84b4f5019e8c8c9e39405f722d1a4706e9"} Nov 03 02:30:02.604491 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:02.604479 4992 scope.go:122] "RemoveContainer" containerID="c3190fb03f66d71d20bf49c25af06e84b4f5019e8c8c9e39405f722d1a4706e9" Nov 03 02:30:03.612328 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.612289 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerStarted","Data":"dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2"} Nov 03 02:30:03.614348 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.614325 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerStarted","Data":"a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa"} Nov 03 02:30:03.616074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.616053 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerStarted","Data":"0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64"} Nov 03 02:30:03.617908 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.617886 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerStarted","Data":"c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62"} Nov 03 02:30:03.619647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.619622 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerStarted","Data":"6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4"} Nov 03 02:30:03.621361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.621342 4992 generic.go:356] "Generic (PLEG): container finished" podID="e2665b1e-0135-4498-b431-768a205a6185" containerID="7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48" exitCode=60 Nov 03 02:30:03.621446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.621422 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" event={"ID":"e2665b1e-0135-4498-b431-768a205a6185","Type":"ContainerDied","Data":"7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48"} Nov 03 02:30:03.621515 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.621491 4992 scope.go:122] "RemoveContainer" containerID="76e0e3aed0b07b1bbce654f1daf35e055e799e81f3498a2624ba486089e94bc8" Nov 03 02:30:03.621621 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.621594 4992 scope.go:122] "RemoveContainer" containerID="7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48" Nov 03 02:30:03.621861 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:03.621836 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2_odh-ai-gateway-infra(e2665b1e-0135-4498-b431-768a205a6185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" podUID="e2665b1e-0135-4498-b431-768a205a6185" Nov 03 02:30:03.732388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.732341 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" podStartSLOduration=3.732327701 podStartE2EDuration="3.732327701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:03.731228822 +0000 UTC m=+4339.310051408" watchObservedRunningTime="2026-11-03 02:30:03.732327701 +0000 UTC m=+4339.311150288" Nov 03 02:30:03.854238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.854183 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" podStartSLOduration=3.854169125 podStartE2EDuration="3.854169125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:03.853030909 +0000 UTC m=+4339.431853496" watchObservedRunningTime="2026-11-03 02:30:03.854169125 +0000 UTC m=+4339.432991712" Nov 03 02:30:03.972836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:03.972782 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" podStartSLOduration=3.972767394 podStartE2EDuration="3.972767394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:03.971230813 +0000 UTC m=+4339.550053400" watchObservedRunningTime="2026-11-03 02:30:03.972767394 +0000 UTC m=+4339.551590009" Nov 03 02:30:04.328939 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.328880 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" podStartSLOduration=4.328865745 podStartE2EDuration="4.328865745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:04.327899802 +0000 UTC m=+4339.906722380" watchObservedRunningTime="2026-11-03 02:30:04.328865745 +0000 UTC m=+4339.907688353" Nov 03 02:30:04.630036 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.629958 4992 generic.go:356] "Generic (PLEG): container finished" podID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerID="dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2" exitCode=60 Nov 03 02:30:04.630466 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.630036 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerDied","Data":"dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2"} Nov 03 02:30:04.630466 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.630097 4992 scope.go:122] "RemoveContainer" containerID="c3190fb03f66d71d20bf49c25af06e84b4f5019e8c8c9e39405f722d1a4706e9" Nov 03 02:30:04.630466 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.630342 4992 scope.go:122] "RemoveContainer" containerID="dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2" Nov 03 02:30:04.630676 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.630654 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks_odh-ai-gateway-infra(de70cdc4-127f-40e4-a3f6-7a65404852a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" Nov 03 02:30:04.632473 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.632448 4992 generic.go:356] "Generic (PLEG): container finished" podID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerID="a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa" exitCode=60 Nov 03 02:30:04.632659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.632480 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerDied","Data":"a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa"} Nov 03 02:30:04.632933 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.632912 4992 scope.go:122] "RemoveContainer" containerID="a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa" Nov 03 02:30:04.633236 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.633212 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h_odh-ai-gateway-infra(946c4ca4-b5cb-495c-ae44-391634a6d663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" Nov 03 02:30:04.634521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.634501 4992 generic.go:356] "Generic (PLEG): container finished" podID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerID="0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64" exitCode=60 Nov 03 02:30:04.634630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.634593 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerDied","Data":"0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64"} Nov 03 02:30:04.634841 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.634823 4992 scope.go:122] "RemoveContainer" containerID="0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64" Nov 03 02:30:04.635153 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.635130 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6_odh-ai-gateway-infra(bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" Nov 03 02:30:04.636738 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.636714 4992 generic.go:356] "Generic (PLEG): container finished" podID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerID="c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62" exitCode=60 Nov 03 02:30:04.636852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.636809 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerDied","Data":"c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62"} Nov 03 02:30:04.637038 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.637017 4992 scope.go:122] "RemoveContainer" containerID="c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62" Nov 03 02:30:04.637319 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.637295 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx_odh-ai-gateway-infra(243c74d3-afa4-4fac-96e7-f57aa76d669c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" Nov 03 02:30:04.638890 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.638870 4992 generic.go:356] "Generic (PLEG): container finished" podID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerID="6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4" exitCode=60 Nov 03 02:30:04.638985 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.638962 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerDied","Data":"6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4"} Nov 03 02:30:04.639171 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.639154 4992 scope.go:122] "RemoveContainer" containerID="6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4" Nov 03 02:30:04.639433 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.639410 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk_odh-ai-gateway-infra(6b2046f0-3311-4f8e-b2b4-56d90838e56c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" Nov 03 02:30:04.640740 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.640724 4992 scope.go:122] "RemoveContainer" containerID="7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48" Nov 03 02:30:04.640952 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:04.640934 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2_odh-ai-gateway-infra(e2665b1e-0135-4498-b431-768a205a6185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" podUID="e2665b1e-0135-4498-b431-768a205a6185" Nov 03 02:30:04.645270 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.645253 4992 scope.go:122] "RemoveContainer" containerID="f6ac49bf49c2a530727768dbad5bc998312ac4a7bd39852739a447d5805e02b6" Nov 03 02:30:04.659215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.659192 4992 scope.go:122] "RemoveContainer" containerID="2a7eea51031ae286d1e7b5c2c7cb41457445c9bee865ddb8fe32bc904c8ab1c8" Nov 03 02:30:04.673288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.673266 4992 scope.go:122] "RemoveContainer" containerID="c28f48c089789cd3bdbda9a14ac01371ec01ffee23abbfc165a40d7658ede386" Nov 03 02:30:04.686849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:04.686825 4992 scope.go:122] "RemoveContainer" containerID="b43f9b2390e7dcbb761d3bbe13fc12d076dcc6886d3b3fa96c213c4dd4dbaac7" Nov 03 02:30:05.648934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:05.648900 4992 scope.go:122] "RemoveContainer" containerID="dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2" Nov 03 02:30:05.649441 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:05.649251 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks_odh-ai-gateway-infra(de70cdc4-127f-40e4-a3f6-7a65404852a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" Nov 03 02:30:05.650691 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:05.650667 4992 scope.go:122] "RemoveContainer" containerID="a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa" Nov 03 02:30:05.651016 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:05.650987 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h_odh-ai-gateway-infra(946c4ca4-b5cb-495c-ae44-391634a6d663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" Nov 03 02:30:05.652430 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:05.652411 4992 scope.go:122] "RemoveContainer" containerID="0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64" Nov 03 02:30:05.652762 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:05.652733 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6_odh-ai-gateway-infra(bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" Nov 03 02:30:05.653854 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:05.653837 4992 scope.go:122] "RemoveContainer" containerID="c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62" Nov 03 02:30:05.654092 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:05.654073 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx_odh-ai-gateway-infra(243c74d3-afa4-4fac-96e7-f57aa76d669c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" Nov 03 02:30:05.655333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:05.655315 4992 scope.go:122] "RemoveContainer" containerID="6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4" Nov 03 02:30:05.655601 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:05.655581 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk_odh-ai-gateway-infra(6b2046f0-3311-4f8e-b2b4-56d90838e56c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" Nov 03 02:30:14.591576 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:14.591532 4992 scope.go:122] "RemoveContainer" containerID="a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa" Nov 03 02:30:15.587808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.587766 4992 scope.go:122] "RemoveContainer" containerID="6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4" Nov 03 02:30:15.587978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.587915 4992 scope.go:122] "RemoveContainer" containerID="0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64" Nov 03 02:30:15.587978 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.587942 4992 scope.go:122] "RemoveContainer" containerID="c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62" Nov 03 02:30:15.789232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.789202 4992 generic.go:356] "Generic (PLEG): container finished" podID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerID="83d318da37232e0611e3c6249a5cd7b3872b50dc56f3b618230a3ef411e5dc7c" exitCode=60 Nov 03 02:30:15.789692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.789279 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerDied","Data":"83d318da37232e0611e3c6249a5cd7b3872b50dc56f3b618230a3ef411e5dc7c"} Nov 03 02:30:15.789692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.789358 4992 scope.go:122] "RemoveContainer" containerID="a9c018094c71f4583f855afb2e5221dd58c1af261baeb0fb445e17f1ec171bfa" Nov 03 02:30:15.789811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:15.789706 4992 scope.go:122] "RemoveContainer" containerID="83d318da37232e0611e3c6249a5cd7b3872b50dc56f3b618230a3ef411e5dc7c" Nov 03 02:30:15.790047 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:15.790027 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h_odh-ai-gateway-infra(946c4ca4-b5cb-495c-ae44-391634a6d663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" Nov 03 02:30:16.588028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.587997 4992 scope.go:122] "RemoveContainer" containerID="dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2" Nov 03 02:30:16.803757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.803718 4992 generic.go:356] "Generic (PLEG): container finished" podID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerID="ebeaff0e9f7771135884ab95582c67e12744ad005349d8099591c92eaaa71eb0" exitCode=60 Nov 03 02:30:16.804264 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.803752 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerDied","Data":"ebeaff0e9f7771135884ab95582c67e12744ad005349d8099591c92eaaa71eb0"} Nov 03 02:30:16.804264 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.803822 4992 scope.go:122] "RemoveContainer" containerID="0a92ace39801ec7602492e5adb014edbfd5926beccf60e08ee778d3ab3489f64" Nov 03 02:30:16.804700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.804681 4992 scope.go:122] "RemoveContainer" containerID="ebeaff0e9f7771135884ab95582c67e12744ad005349d8099591c92eaaa71eb0" Nov 03 02:30:16.805101 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:16.805071 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6_odh-ai-gateway-infra(bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" Nov 03 02:30:16.806677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.806628 4992 generic.go:356] "Generic (PLEG): container finished" podID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerID="51cac03c52930b38ed2a0717bd7486e92878b477c7886ff5aa6b5163c0bf1fed" exitCode=60 Nov 03 02:30:16.806754 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.806731 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerDied","Data":"51cac03c52930b38ed2a0717bd7486e92878b477c7886ff5aa6b5163c0bf1fed"} Nov 03 02:30:16.807142 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.807123 4992 scope.go:122] "RemoveContainer" containerID="51cac03c52930b38ed2a0717bd7486e92878b477c7886ff5aa6b5163c0bf1fed" Nov 03 02:30:16.807451 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:16.807430 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx_odh-ai-gateway-infra(243c74d3-afa4-4fac-96e7-f57aa76d669c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" Nov 03 02:30:16.837023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.836987 4992 generic.go:356] "Generic (PLEG): container finished" podID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerID="01779c38e75f1fdc858a600fbad32b0ce872b3bdd9737d99bb5388f1ef298c18" exitCode=60 Nov 03 02:30:16.837200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.837015 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerDied","Data":"01779c38e75f1fdc858a600fbad32b0ce872b3bdd9737d99bb5388f1ef298c18"} Nov 03 02:30:16.837408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.837388 4992 scope.go:122] "RemoveContainer" containerID="01779c38e75f1fdc858a600fbad32b0ce872b3bdd9737d99bb5388f1ef298c18" Nov 03 02:30:16.837685 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:16.837649 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk_odh-ai-gateway-infra(6b2046f0-3311-4f8e-b2b4-56d90838e56c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" Nov 03 02:30:16.841410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.841390 4992 scope.go:122] "RemoveContainer" containerID="c3e64d20dbd8a35bc7431f1d62f715a4da3988e9425780a258c841a1412c3d62" Nov 03 02:30:16.855004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.854977 4992 scope.go:122] "RemoveContainer" containerID="6e10f16e34e7e93ab5bf9e9523c908b65c8e54f759bf4f05012a05a9ed3c99b4" Nov 03 02:30:16.925294 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:16.924967 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:17.091792 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.091723 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:17.137462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.137427 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\" (UniqueName: \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\") pod \"946c4ca4-b5cb-495c-ae44-391634a6d663\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " Nov 03 02:30:17.137681 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.137490 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\") pod \"946c4ca4-b5cb-495c-ae44-391634a6d663\" (UID: \"946c4ca4-b5cb-495c-ae44-391634a6d663\") " Nov 03 02:30:17.137982 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.137944 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca" pod "946c4ca4-b5cb-495c-ae44-391634a6d663" (UID: "946c4ca4-b5cb-495c-ae44-391634a6d663"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:17.138116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.138085 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946c4ca4-b5cb-495c-ae44-391634a6d663-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.140035 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.140011 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d" pod "946c4ca4-b5cb-495c-ae44-391634a6d663" (UID: "946c4ca4-b5cb-495c-ae44-391634a6d663"). InnerVolumeSpecName "kube-api-access-fbc7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:17.239259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.239222 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbc7d\" (UniqueName: \"kubernetes.io/projected/946c4ca4-b5cb-495c-ae44-391634a6d663-kube-api-access-fbc7d\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:17.587779 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.587742 4992 scope.go:122] "RemoveContainer" containerID="7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48" Nov 03 02:30:17.855133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.855045 4992 generic.go:356] "Generic (PLEG): container finished" podID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerID="58357624bd248ad0a074281c5908c0cbdb2cb0afb53d1ad322fd1ca0bffa8415" exitCode=60 Nov 03 02:30:17.855133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.855113 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerDied","Data":"58357624bd248ad0a074281c5908c0cbdb2cb0afb53d1ad322fd1ca0bffa8415"} Nov 03 02:30:17.855694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.855175 4992 scope.go:122] "RemoveContainer" containerID="dcb0033785c1d13acc2a790f7bc5d270a9d263396428e238e72d34bfcd47a3b2" Nov 03 02:30:17.855694 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.855578 4992 scope.go:122] "RemoveContainer" containerID="58357624bd248ad0a074281c5908c0cbdb2cb0afb53d1ad322fd1ca0bffa8415" Nov 03 02:30:17.855891 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:17.855866 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks_odh-ai-gateway-infra(de70cdc4-127f-40e4-a3f6-7a65404852a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" Nov 03 02:30:17.856606 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.856586 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" event={"ID":"946c4ca4-b5cb-495c-ae44-391634a6d663","Type":"ContainerDied","Data":"487c80cefb40332fc8acb8cd048ca81ba1c4e946f9d42928ef74bc436fa31b79"} Nov 03 02:30:17.856606 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.856601 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h" Nov 03 02:30:17.869393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.869371 4992 scope.go:122] "RemoveContainer" containerID="83d318da37232e0611e3c6249a5cd7b3872b50dc56f3b618230a3ef411e5dc7c" Nov 03 02:30:17.936229 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:17.936195 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:18.128989 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.128951 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:18.131852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.131825 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e051f3-29894550-v8x4h"] Nov 03 02:30:18.135774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.135750 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:18.188360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.188322 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:18.268347 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.268315 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\" (UniqueName: \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\") pod \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " Nov 03 02:30:18.268548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.268508 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\") pod \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\" (UID: \"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43\") " Nov 03 02:30:18.268985 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.268861 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca" pod "bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" (UID: "bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.269105 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.269045 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.272089 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.272058 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr" pod "bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" (UID: "bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43"). InnerVolumeSpecName "kube-api-access-x5prr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:18.296176 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.296139 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:18.320762 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.320733 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:18.371307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.369602 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\") pod \"243c74d3-afa4-4fac-96e7-f57aa76d669c\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " Nov 03 02:30:18.371307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.369807 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\" (UniqueName: \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\") pod \"243c74d3-afa4-4fac-96e7-f57aa76d669c\" (UID: \"243c74d3-afa4-4fac-96e7-f57aa76d669c\") " Nov 03 02:30:18.371307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.370268 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5prr\" (UniqueName: \"kubernetes.io/projected/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43-kube-api-access-x5prr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.371307 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.371028 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca" pod "243c74d3-afa4-4fac-96e7-f57aa76d669c" (UID: "243c74d3-afa4-4fac-96e7-f57aa76d669c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.373355 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.373324 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj" pod "243c74d3-afa4-4fac-96e7-f57aa76d669c" (UID: "243c74d3-afa4-4fac-96e7-f57aa76d669c"). InnerVolumeSpecName "kube-api-access-lzwhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:18.443783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.443752 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:18.471606 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.471571 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\") pod \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " Nov 03 02:30:18.471863 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.471840 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca" pod "6b2046f0-3311-4f8e-b2b4-56d90838e56c" (UID: "6b2046f0-3311-4f8e-b2b4-56d90838e56c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:18.471957 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.471871 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\" (UniqueName: \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\") pod \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\" (UID: \"6b2046f0-3311-4f8e-b2b4-56d90838e56c\") " Nov 03 02:30:18.472225 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.472207 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/243c74d3-afa4-4fac-96e7-f57aa76d669c-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.472287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.472229 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b2046f0-3311-4f8e-b2b4-56d90838e56c-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.472287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.472246 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwhj\" (UniqueName: \"kubernetes.io/projected/243c74d3-afa4-4fac-96e7-f57aa76d669c-kube-api-access-lzwhj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.473958 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.473938 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j" pod "6b2046f0-3311-4f8e-b2b4-56d90838e56c" (UID: "6b2046f0-3311-4f8e-b2b4-56d90838e56c"). InnerVolumeSpecName "kube-api-access-sxt5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:18.573344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.573314 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxt5j\" (UniqueName: \"kubernetes.io/projected/6b2046f0-3311-4f8e-b2b4-56d90838e56c-kube-api-access-sxt5j\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:18.595206 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.595162 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" path="/var/lib/kubelet/pods/946c4ca4-b5cb-495c-ae44-391634a6d663/volumes" Nov 03 02:30:18.864386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.864357 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" Nov 03 02:30:18.864386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.864370 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6" event={"ID":"bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43","Type":"ContainerDied","Data":"6be950e49b822a726bb7838aa678fb48fae076d62c2f3a3448684bb3373e8ab3"} Nov 03 02:30:18.864970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.864436 4992 scope.go:122] "RemoveContainer" containerID="ebeaff0e9f7771135884ab95582c67e12744ad005349d8099591c92eaaa71eb0" Nov 03 02:30:18.866003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.865974 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" Nov 03 02:30:18.866161 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.865998 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx" event={"ID":"243c74d3-afa4-4fac-96e7-f57aa76d669c","Type":"ContainerDied","Data":"adcdac0522418de6756c0196e5a2e3972aad7b19708f53c9d6b436cccd8161c2"} Nov 03 02:30:18.867647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.867545 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" event={"ID":"6b2046f0-3311-4f8e-b2b4-56d90838e56c","Type":"ContainerDied","Data":"1e19d65c2f859e04bc09cb3baad22e0a8deb18499bb216e80c6d542a9a4abc82"} Nov 03 02:30:18.867741 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.867646 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk" Nov 03 02:30:18.870480 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.870452 4992 generic.go:356] "Generic (PLEG): container finished" podID="e2665b1e-0135-4498-b431-768a205a6185" containerID="76d0499bf6fb52c1f244e70f1c86e958f78af2ba9c5d1ed51fcd2f94f2edad41" exitCode=60 Nov 03 02:30:18.870631 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.870514 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" event={"ID":"e2665b1e-0135-4498-b431-768a205a6185","Type":"ContainerDied","Data":"76d0499bf6fb52c1f244e70f1c86e958f78af2ba9c5d1ed51fcd2f94f2edad41"} Nov 03 02:30:18.870956 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.870916 4992 scope.go:122] "RemoveContainer" containerID="76d0499bf6fb52c1f244e70f1c86e958f78af2ba9c5d1ed51fcd2f94f2edad41" Nov 03 02:30:18.871228 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:18.871200 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2_odh-ai-gateway-infra(e2665b1e-0135-4498-b431-768a205a6185)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" podUID="e2665b1e-0135-4498-b431-768a205a6185" Nov 03 02:30:18.877314 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.877292 4992 scope.go:122] "RemoveContainer" containerID="51cac03c52930b38ed2a0717bd7486e92878b477c7886ff5aa6b5163c0bf1fed" Nov 03 02:30:18.886246 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.886228 4992 scope.go:122] "RemoveContainer" containerID="01779c38e75f1fdc858a600fbad32b0ce872b3bdd9737d99bb5388f1ef298c18" Nov 03 02:30:18.896955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.896934 4992 scope.go:122] "RemoveContainer" containerID="7e676452ad24c11d0c6ce99cfb7ba6d0c759cd5c647f18683d92bbabdab17b48" Nov 03 02:30:18.986004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:18.985903 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:19.157539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.157504 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:19.162233 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.162195 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6a5d73-29894550-w7dwk"] Nov 03 02:30:19.185181 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.185155 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:19.276970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.276866 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:19.280433 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.280401 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894550-dpkh6"] Nov 03 02:30:19.282968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.282942 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\" (UniqueName: \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\") pod \"de70cdc4-127f-40e4-a3f6-7a65404852a2\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " Nov 03 02:30:19.283098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.283001 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\") pod \"de70cdc4-127f-40e4-a3f6-7a65404852a2\" (UID: \"de70cdc4-127f-40e4-a3f6-7a65404852a2\") " Nov 03 02:30:19.283626 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.283467 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca" pod "de70cdc4-127f-40e4-a3f6-7a65404852a2" (UID: "de70cdc4-127f-40e4-a3f6-7a65404852a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:19.287833 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.287807 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2" pod "de70cdc4-127f-40e4-a3f6-7a65404852a2" (UID: "de70cdc4-127f-40e4-a3f6-7a65404852a2"). InnerVolumeSpecName "kube-api-access-vdbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:19.384906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.384865 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de70cdc4-127f-40e4-a3f6-7a65404852a2-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.384906 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.384902 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbt2\" (UniqueName: \"kubernetes.io/projected/de70cdc4-127f-40e4-a3f6-7a65404852a2-kube-api-access-vdbt2\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:19.393877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.393841 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:19.402015 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.401979 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98a394-29894550-2l9jx"] Nov 03 02:30:19.889034 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.888991 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" event={"ID":"de70cdc4-127f-40e4-a3f6-7a65404852a2","Type":"ContainerDied","Data":"bb5d52edd43cc05afdfb0a6837c1d01dbccb8687eda840d761a4b97c5b109393"} Nov 03 02:30:19.889545 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.889055 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks" Nov 03 02:30:19.889545 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:19.889071 4992 scope.go:122] "RemoveContainer" containerID="58357624bd248ad0a074281c5908c0cbdb2cb0afb53d1ad322fd1ca0bffa8415" Nov 03 02:30:20.050207 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.050163 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:20.064499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.064465 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:20.103313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.103275 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a54cab-29894550-45tks"] Nov 03 02:30:20.220525 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.220496 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:20.295731 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.295692 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\") pod \"e2665b1e-0135-4498-b431-768a205a6185\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " Nov 03 02:30:20.295939 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.295923 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\" (UniqueName: \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\") pod \"e2665b1e-0135-4498-b431-768a205a6185\" (UID: \"e2665b1e-0135-4498-b431-768a205a6185\") " Nov 03 02:30:20.296184 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.296158 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca" pod "e2665b1e-0135-4498-b431-768a205a6185" (UID: "e2665b1e-0135-4498-b431-768a205a6185"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:30:20.296412 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.296391 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2665b1e-0135-4498-b431-768a205a6185-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.298017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.297988 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd" pod "e2665b1e-0135-4498-b431-768a205a6185" (UID: "e2665b1e-0135-4498-b431-768a205a6185"). InnerVolumeSpecName "kube-api-access-qs9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:30:20.398363 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.398328 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9dd\" (UniqueName: \"kubernetes.io/projected/e2665b1e-0135-4498-b431-768a205a6185-kube-api-access-qs9dd\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:30:20.595862 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.595778 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" path="/var/lib/kubelet/pods/243c74d3-afa4-4fac-96e7-f57aa76d669c/volumes" Nov 03 02:30:20.596248 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.596234 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" path="/var/lib/kubelet/pods/6b2046f0-3311-4f8e-b2b4-56d90838e56c/volumes" Nov 03 02:30:20.596650 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.596636 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" path="/var/lib/kubelet/pods/bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43/volumes" Nov 03 02:30:20.597321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.597306 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" path="/var/lib/kubelet/pods/de70cdc4-127f-40e4-a3f6-7a65404852a2/volumes" Nov 03 02:30:20.898185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.898091 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" Nov 03 02:30:20.898185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.898118 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2" event={"ID":"e2665b1e-0135-4498-b431-768a205a6185","Type":"ContainerDied","Data":"d997a4823c5c362e36212e427dbd3b94d20bceb07d68acf423b5f945bf8754e6"} Nov 03 02:30:20.898767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:20.898189 4992 scope.go:122] "RemoveContainer" containerID="76d0499bf6fb52c1f244e70f1c86e958f78af2ba9c5d1ed51fcd2f94f2edad41" Nov 03 02:30:21.073496 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:21.073450 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:21.082881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:21.082845 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b6c56b-29894550-xdvl2"] Nov 03 02:30:22.595462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:30:22.595416 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2665b1e-0135-4498-b431-768a205a6185" path="/var/lib/kubelet/pods/e2665b1e-0135-4498-b431-768a205a6185/volumes" Nov 03 02:30:32.152668 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:30:32.152627 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:31:02.152072 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:31:02.152031 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:31:11.247649 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:11.247608 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:31:11.248337 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:11.247826 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" containerID="cri-o://702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30" gracePeriod=30 Nov 03 02:31:11.672031 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:11.671995 4992 patch_prober.go:29] interesting pod/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:31:11.672214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:11.672061 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:31:16.522238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.522154 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:31:16.654914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.654870 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.654934 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655018 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655131 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655171 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655443 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655231 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:16.655443 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655243 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655443 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655330 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:16.655443 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655430 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655443 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655436 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:16.655793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655466 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655531 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\") pod \"2f242d8f-e746-48ef-a383-9e06f3565481\" (UID: \"2f242d8f-e746-48ef-a383-9e06f3565481\") " Nov 03 02:31:16.655793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655681 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:16.656010 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.655984 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:16.656306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.656207 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.656306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.656233 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.656306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.656249 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f242d8f-e746-48ef-a383-9e06f3565481-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.656306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.656262 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.656306 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.656277 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.657462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.657436 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:31:16.657942 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.657916 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:16.658142 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.658119 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:31:16.658212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.658178 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n" pod "2f242d8f-e746-48ef-a383-9e06f3565481" (UID: "2f242d8f-e746-48ef-a383-9e06f3565481"). InnerVolumeSpecName "kube-api-access-sf47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:16.758083 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.758042 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f242d8f-e746-48ef-a383-9e06f3565481-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.758083 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.758076 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f242d8f-e746-48ef-a383-9e06f3565481-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.758083 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.758087 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf47n\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-kube-api-access-sf47n\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:16.758345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:16.758097 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f242d8f-e746-48ef-a383-9e06f3565481-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:17.323043 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.323009 4992 generic.go:356] "Generic (PLEG): container finished" podID="2f242d8f-e746-48ef-a383-9e06f3565481" containerID="702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30" exitCode=0 Nov 03 02:31:17.323239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.323067 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" Nov 03 02:31:17.323239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.323092 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerDied","Data":"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30"} Nov 03 02:31:17.323239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.323144 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz" event={"ID":"2f242d8f-e746-48ef-a383-9e06f3565481","Type":"ContainerDied","Data":"5b6672cfb02b0c388284a2b2a8d8b67975c8aaf4178e670f6082394dd35c3e1c"} Nov 03 02:31:17.323239 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.323166 4992 scope.go:122] "RemoveContainer" containerID="702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30" Nov 03 02:31:17.334535 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.334505 4992 scope.go:122] "RemoveContainer" containerID="5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218" Nov 03 02:31:17.344044 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.344018 4992 scope.go:122] "RemoveContainer" containerID="702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30" Nov 03 02:31:17.344344 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:31:17.344317 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30\": container with ID starting with 702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30 not found: ID does not exist" containerID="702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30" Nov 03 02:31:17.344398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.344355 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30"} err="failed to get container status \"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30\": rpc error: code = NotFound desc = could not find container \"702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30\": container with ID starting with 702528173de05bafaa1991ea9472cda2d096b0a745ecd710cf0e7c8a86507a30 not found: ID does not exist" Nov 03 02:31:17.344398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.344376 4992 scope.go:122] "RemoveContainer" containerID="5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218" Nov 03 02:31:17.344687 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:31:17.344668 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218\": container with ID starting with 5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218 not found: ID does not exist" containerID="5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218" Nov 03 02:31:17.344758 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.344691 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218"} err="failed to get container status \"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218\": rpc error: code = NotFound desc = could not find container \"5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218\": container with ID starting with 5ea7ca0b6a0f17cca0fbc920f913239a656105baf4353c0c034dcf176ce1c218 not found: ID does not exist" Nov 03 02:31:17.546834 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.546797 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:31:17.599250 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:17.599173 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e051f3-openshift-default-57479d6d5f-rr7kz"] Nov 03 02:31:18.595155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:18.595117 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" path="/var/lib/kubelet/pods/2f242d8f-e746-48ef-a383-9e06f3565481/volumes" Nov 03 02:31:25.343700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.343663 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:31:25.344131 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.343866 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" podUID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" containerName="maas-api" containerID="cri-o://06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057" gracePeriod=30 Nov 03 02:31:25.673058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.673025 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:31:25.783792 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.783759 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\") pod \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " Nov 03 02:31:25.783985 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.783806 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\" (UniqueName: \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\") pod \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " Nov 03 02:31:25.783985 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.783904 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") pod \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " Nov 03 02:31:25.784104 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.784078 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\") pod \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\" (UID: \"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35\") " Nov 03 02:31:25.784526 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.784494 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca" pod "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" (UID: "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:31:25.784837 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.784811 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:25.786404 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.786300 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs" pod "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" (UID: "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:25.786587 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.786523 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls" pod "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" (UID: "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:31:25.786698 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.786633 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj" pod "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" (UID: "0e8030fa-2d3c-4e8f-a2d4-b47eed643e35"). InnerVolumeSpecName "kube-api-access-n4vdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:31:25.886068 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.885986 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:25.886068 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.886019 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vdj\" (UniqueName: \"kubernetes.io/projected/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-kube-api-access-n4vdj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:25.886068 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:25.886028 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:31:26.391351 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.391315 4992 generic.go:356] "Generic (PLEG): container finished" podID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" containerID="06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057" exitCode=0 Nov 03 02:31:26.391829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.391389 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" Nov 03 02:31:26.391829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.391403 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" event={"ID":"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35","Type":"ContainerDied","Data":"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057"} Nov 03 02:31:26.391829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.391478 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm" event={"ID":"0e8030fa-2d3c-4e8f-a2d4-b47eed643e35","Type":"ContainerDied","Data":"d9f8c5f6c76e6c4e7fb4bdccd6f61a4f4dffbcc2893bfbe301d466c9e94c83dc"} Nov 03 02:31:26.391829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.391501 4992 scope.go:122] "RemoveContainer" containerID="06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057" Nov 03 02:31:26.402141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.402117 4992 scope.go:122] "RemoveContainer" containerID="06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057" Nov 03 02:31:26.402439 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:31:26.402421 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057\": container with ID starting with 06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057 not found: ID does not exist" containerID="06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057" Nov 03 02:31:26.402492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.402450 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057"} err="failed to get container status \"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057\": rpc error: code = NotFound desc = could not find container \"06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057\": container with ID starting with 06ee0e3290beca36bc66ad3b79a0cc6fe6c7ad8f7f652a26234a2522ebc3c057 not found: ID does not exist" Nov 03 02:31:26.562074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.562037 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:31:26.613740 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:26.613707 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b6c56b-6b984cffcc-bg4dm"] Nov 03 02:31:28.595312 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:31:28.595279 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" path="/var/lib/kubelet/pods/0e8030fa-2d3c-4e8f-a2d4-b47eed643e35/volumes" Nov 03 02:31:32.150987 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:31:32.150949 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:32:02.151522 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:32:02.151486 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:32:32.152682 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:32:32.152640 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:32:45.257481 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:32:45.257451 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:32:45.262183 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:32:45.262155 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:33:02.151185 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:33:02.151147 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:33:32.154237 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:33:32.154143 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:34:02.152628 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:34:02.152585 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:34:32.151250 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:34:32.151207 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:34:52.791358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791321 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:34:52.791909 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791891 4992 cpu_manager.go:414] "containerMap: removing container" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791912 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791926 4992 cpu_manager.go:414] "containerMap: removing container" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791931 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791939 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791943 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791950 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:34:52.791954 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791955 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791967 4992 cpu_manager.go:414] "containerMap: removing container" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791971 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791978 4992 cpu_manager.go:414] "containerMap: removing container" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791982 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791988 4992 cpu_manager.go:414] "containerMap: removing container" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.791992 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792002 4992 cpu_manager.go:414] "containerMap: removing container" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792007 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792014 4992 cpu_manager.go:414] "containerMap: removing container" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792018 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792024 4992 cpu_manager.go:414] "containerMap: removing container" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792028 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792033 4992 cpu_manager.go:414] "containerMap: removing container" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792037 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792055 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792060 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792067 4992 cpu_manager.go:414] "containerMap: removing container" podUID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" containerName="maas-api" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792072 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" containerName="maas-api" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792080 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792084 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792088 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792093 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792102 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792106 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792114 4992 cpu_manager.go:414] "containerMap: removing container" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.792168 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792118 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792228 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792236 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792242 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792251 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792257 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792263 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792268 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792275 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="de70cdc4-127f-40e4-a3f6-7a65404852a2" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792281 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792286 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792291 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792296 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792303 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792309 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8030fa-2d3c-4e8f-a2d4-b47eed643e35" containerName="maas-api" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792316 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792320 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:34:52.793003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.792325 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2665b1e-0135-4498-b431-768a205a6185" containerName="cleanup" Nov 03 02:34:52.795352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.795335 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.850118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.850084 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:34:52.898119 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.898085 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"openshift-service-ca.crt\"" Nov 03 02:34:52.901612 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.901550 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"kube-root-ca.crt\"" Nov 03 02:34:52.901769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.901610 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"rate-test-model-6a5d73-kserve-self-signed-certs\"" Nov 03 02:34:52.901769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.901710 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"default-dockercfg-hgwfm\"" Nov 03 02:34:52.929618 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.929542 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.929797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.929700 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.929797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.929742 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.929886 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.929863 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.929919 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.929897 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86p8q\" (UniqueName: \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:52.930042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:52.930025 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031032 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.030991 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86p8q\" (UniqueName: \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031107 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031188 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031300 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031239 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031339 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031299 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031393 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031366 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031620 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031592 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031662 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.031813 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.031775 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.033535 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.033506 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.033847 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.033828 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.097000 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.096907 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86p8q\" (UniqueName: \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\") pod \"rate-test-model-6a5d73-kserve-55f958589-mxvnw\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.109018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.108987 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:34:53.496008 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:34:53.495977 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda66dc16f_e1a9_4462_845d_3c5b35e86ce7.slice/crio-5aeb9f7f64c67321b0a1da044f4c53fa45dc4b15520a32dc1b832152fdf2eeb6 WatchSource:0}: Error finding container 5aeb9f7f64c67321b0a1da044f4c53fa45dc4b15520a32dc1b832152fdf2eeb6: Status 404 returned error can't find the container with id 5aeb9f7f64c67321b0a1da044f4c53fa45dc4b15520a32dc1b832152fdf2eeb6 Nov 03 02:34:53.499219 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.499197 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:34:53.737252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.737204 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" event={"ID":"a66dc16f-e1a9-4462-845d-3c5b35e86ce7","Type":"ContainerStarted","Data":"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a"} Nov 03 02:34:53.737252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.737257 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" event={"ID":"a66dc16f-e1a9-4462-845d-3c5b35e86ce7","Type":"ContainerStarted","Data":"5aeb9f7f64c67321b0a1da044f4c53fa45dc4b15520a32dc1b832152fdf2eeb6"} Nov 03 02:34:53.860820 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:53.860694 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" podStartSLOduration=1.86067662 podStartE2EDuration="1.86067662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:53.858427777 +0000 UTC m=+4629.437250363" watchObservedRunningTime="2026-11-03 02:34:53.86067662 +0000 UTC m=+4629.439499207" Nov 03 02:34:54.110006 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:34:54.109974 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:35:02.152337 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:35:02.152251 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:35:03.109260 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:03.109223 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:35:03.122930 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:03.122896 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:35:03.815521 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:03.815486 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:35:17.541146 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:17.541097 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:35:17.541637 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:17.541388 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" containerID="cri-o://27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27" gracePeriod=30 Nov 03 02:35:22.808133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.808061 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:35:22.925672 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.925628 4992 generic.go:356] "Generic (PLEG): container finished" podID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerID="27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27" exitCode=0 Nov 03 02:35:22.925840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.925684 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" Nov 03 02:35:22.925840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.925712 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerDied","Data":"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27"} Nov 03 02:35:22.925840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.925773 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj" event={"ID":"e13cbd9f-652a-4eb3-872f-382758aca6b5","Type":"ContainerDied","Data":"3df3d075d9f1b7e676f3600ba82858818d2774f225072d8b206a7f12aaca423b"} Nov 03 02:35:22.925840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.925797 4992 scope.go:122] "RemoveContainer" containerID="27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27" Nov 03 02:35:22.938117 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.938092 4992 scope.go:122] "RemoveContainer" containerID="f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed" Nov 03 02:35:22.942663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942641 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.942775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942675 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.942775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942701 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.942775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942719 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.942775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942766 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.942979 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942865 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.943045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942981 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.943045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.942976 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:22.943157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943041 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:35:22.943157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943113 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.943157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943152 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\") pod \"e13cbd9f-652a-4eb3-872f-382758aca6b5\" (UID: \"e13cbd9f-652a-4eb3-872f-382758aca6b5\") " Nov 03 02:35:22.943380 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943335 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:22.943475 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943438 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:22.943632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943613 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:22.943865 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943841 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:22.943948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943870 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13cbd9f-652a-4eb3-872f-382758aca6b5-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:22.943948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943885 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:22.943948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943899 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:22.943948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.943913 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:22.945094 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.945063 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "kube-api-access-p4pzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:22.945644 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.945614 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:35:22.945878 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.945851 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:35:22.945948 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.945891 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo" pod "e13cbd9f-652a-4eb3-872f-382758aca6b5" (UID: "e13cbd9f-652a-4eb3-872f-382758aca6b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:35:22.948867 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.948846 4992 scope.go:122] "RemoveContainer" containerID="27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27" Nov 03 02:35:22.949195 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:35:22.949171 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27\": container with ID starting with 27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27 not found: ID does not exist" containerID="27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27" Nov 03 02:35:22.949269 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.949206 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27"} err="failed to get container status \"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27\": rpc error: code = NotFound desc = could not find container \"27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27\": container with ID starting with 27c28e0d06d1aaa43f3371dd448076b292f6db056e7c35a72162736f5788ed27 not found: ID does not exist" Nov 03 02:35:22.949269 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.949227 4992 scope.go:122] "RemoveContainer" containerID="f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed" Nov 03 02:35:22.949499 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:35:22.949463 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed\": container with ID starting with f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed not found: ID does not exist" containerID="f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed" Nov 03 02:35:22.949499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:22.949488 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed"} err="failed to get container status \"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed\": rpc error: code = NotFound desc = could not find container \"f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed\": container with ID starting with f5db74b2ca61b95a9574d568c644a4830c2b045eb25c38417951104c0fc7e7ed not found: ID does not exist" Nov 03 02:35:23.045668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.045611 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:23.045668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.045644 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:23.045668 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.045657 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pzb\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-kube-api-access-p4pzb\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:23.045991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.045681 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13cbd9f-652a-4eb3-872f-382758aca6b5-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:35:23.354286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.354219 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:35:23.356941 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:23.356915 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-1c9b4e-openshift-default-54cf99fcbc-2lhhj"] Nov 03 02:35:24.595169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:35:24.595133 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" path="/var/lib/kubelet/pods/e13cbd9f-652a-4eb3-872f-382758aca6b5/volumes" Nov 03 02:35:32.151230 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:35:32.151185 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:36:02.151880 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:36:02.151836 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:36:11.777460 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:11.777414 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 02:36:12.228974 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.228929 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" podUID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" containerName="maas-api" containerID="cri-o://861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5" gracePeriod=30 Nov 03 02:36:12.493925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.493859 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 02:36:12.578878 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.578836 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\") pod \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " Nov 03 02:36:12.579057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.578921 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\" (UniqueName: \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\") pod \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " Nov 03 02:36:12.579147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.579117 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\") pod \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " Nov 03 02:36:12.579215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.579168 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") pod \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\" (UID: \"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b\") " Nov 03 02:36:12.579215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.579187 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca" pod "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" (UID: "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:36:12.579479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.579464 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:12.581216 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.581181 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs" pod "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" (UID: "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:36:12.581335 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.581217 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls" pod "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" (UID: "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:36:12.581335 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.581262 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv" pod "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" (UID: "8d0729db-abdb-4e14-9d5f-cc2efc1ef72b"). InnerVolumeSpecName "kube-api-access-gl8kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:36:12.680451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.680414 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:12.680451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.680447 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:12.680451 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:12.680458 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl8kv\" (UniqueName: \"kubernetes.io/projected/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b-kube-api-access-gl8kv\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:13.237005 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.236964 4992 generic.go:356] "Generic (PLEG): container finished" podID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" containerID="861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5" exitCode=0 Nov 03 02:36:13.237519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.237015 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" Nov 03 02:36:13.237519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.237045 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" event={"ID":"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b","Type":"ContainerDied","Data":"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5"} Nov 03 02:36:13.237519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.237107 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np" event={"ID":"8d0729db-abdb-4e14-9d5f-cc2efc1ef72b","Type":"ContainerDied","Data":"6594f6907f79715243cd5f41132dfebd1716eeaed31092c47bf77c4a3dba609f"} Nov 03 02:36:13.237519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.237121 4992 scope.go:122] "RemoveContainer" containerID="861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5" Nov 03 02:36:13.246918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.246895 4992 scope.go:122] "RemoveContainer" containerID="861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5" Nov 03 02:36:13.247207 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:36:13.247187 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5\": container with ID starting with 861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5 not found: ID does not exist" containerID="861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5" Nov 03 02:36:13.247259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.247222 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5"} err="failed to get container status \"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5\": rpc error: code = NotFound desc = could not find container \"861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5\": container with ID starting with 861f82a2d49e021191c337db9745935d3c9b135037c9252e15147bed3771a8d5 not found: ID does not exist" Nov 03 02:36:13.346660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.346620 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:36:13.347190 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347173 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" containerName="maas-api" Nov 03 02:36:13.347190 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347190 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" containerName="maas-api" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347199 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347204 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347212 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347217 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347230 4992 cpu_manager.go:414] "containerMap: removing container" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347236 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243c74d3-afa4-4fac-96e7-f57aa76d669c" containerName="cleanup" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347248 4992 cpu_manager.go:414] "containerMap: removing container" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347255 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347274 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347278 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347285 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:36:13.347290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347290 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347381 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347392 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebd92a5-ae5d-4a4e-8e62-0d0beeba8d43" containerName="cleanup" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347397 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2046f0-3311-4f8e-b2b4-56d90838e56c" containerName="cleanup" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347406 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" containerName="maas-api" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347412 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f242d8f-e746-48ef-a383-9e06f3565481" containerName="istio-proxy" Nov 03 02:36:13.347677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.347420 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="946c4ca4-b5cb-495c-ae44-391634a6d663" containerName="cleanup" Nov 03 02:36:13.352159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.352139 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.378798 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.378763 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:36:13.406590 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.406539 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f0391a-serving-cert\"" Nov 03 02:36:13.410760 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.410730 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 02:36:13.414902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.414875 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e1793d-69cbf65f47-982np"] Nov 03 02:36:13.491343 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.491260 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.491343 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.491304 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skvc\" (UniqueName: \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.491343 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.491331 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.491620 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.491512 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.592763 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.592719 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.592946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.592788 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skvc\" (UniqueName: \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.592946 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.592825 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.593095 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.592989 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.593854 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.593831 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.595483 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.595457 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.595584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.595542 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.655830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.655791 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skvc\" (UniqueName: \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\") pod \"maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:13.666226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:13.666195 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:14.059333 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:36:14.059305 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6457b78d_7602_4ad4_aae5_634f1579c39b.slice/crio-a496003374ed7d3d4871afd2036a50fd75a9a3622dcb64de58b98c9040ef9c5e WatchSource:0}: Error finding container a496003374ed7d3d4871afd2036a50fd75a9a3622dcb64de58b98c9040ef9c5e: Status 404 returned error can't find the container with id a496003374ed7d3d4871afd2036a50fd75a9a3622dcb64de58b98c9040ef9c5e Nov 03 02:36:14.061184 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:14.061160 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 02:36:14.061404 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:14.061382 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:36:14.245022 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:14.244984 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" event={"ID":"6457b78d-7602-4ad4-aae5-634f1579c39b","Type":"ContainerStarted","Data":"a496003374ed7d3d4871afd2036a50fd75a9a3622dcb64de58b98c9040ef9c5e"} Nov 03 02:36:14.595084 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:14.595051 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0729db-abdb-4e14-9d5f-cc2efc1ef72b" path="/var/lib/kubelet/pods/8d0729db-abdb-4e14-9d5f-cc2efc1ef72b/volumes" Nov 03 02:36:15.253295 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:15.253191 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" event={"ID":"6457b78d-7602-4ad4-aae5-634f1579c39b","Type":"ContainerStarted","Data":"75a764ba2c6b6b51fccb732c03cb6b1edd829026a9058f674fce80eb03ada0e6"} Nov 03 02:36:15.253295 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:15.253263 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:15.374693 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:15.374630 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" podStartSLOduration=1.460284454 podStartE2EDuration="2.37461146s" totalImagesPullingTime="914.327006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:36:15.374111223 +0000 UTC m=+4710.952933810" watchObservedRunningTime="2026-11-03 02:36:15.37461146 +0000 UTC m=+4710.953434047" Nov 03 02:36:18.849491 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:18.849451 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 02:36:18.850020 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:18.849742 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" containerID="cri-o://2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b" gracePeriod=30 Nov 03 02:36:21.268672 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:21.268638 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:36:24.127709 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.127645 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 02:36:24.235596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235496 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.235596 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235577 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.235826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235633 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.235826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235682 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.235826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235746 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.235826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235782 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.236055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235831 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.236055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235904 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.236055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235935 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.236055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.235977 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\") pod \"af1bf105-6ac1-4d23-b5ce-904d17e86037\" (UID: \"af1bf105-6ac1-4d23-b5ce-904d17e86037\") " Nov 03 02:36:24.236271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236063 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.236271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236118 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:36:24.236271 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236225 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.236457 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236429 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.236783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236649 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.236783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236679 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.236783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236696 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af1bf105-6ac1-4d23-b5ce-904d17e86037-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.236783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236711 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.236783 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.236724 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.238692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.238666 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:36:24.239202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.238956 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:36:24.239321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.239295 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:36:24.239827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.239788 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479" pod "af1bf105-6ac1-4d23-b5ce-904d17e86037" (UID: "af1bf105-6ac1-4d23-b5ce-904d17e86037"). InnerVolumeSpecName "kube-api-access-gh479". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:36:24.328526 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.328490 4992 generic.go:356] "Generic (PLEG): container finished" podID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerID="2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b" exitCode=0 Nov 03 02:36:24.328729 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.328532 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" event={"ID":"af1bf105-6ac1-4d23-b5ce-904d17e86037","Type":"ContainerDied","Data":"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b"} Nov 03 02:36:24.328729 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.328595 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" Nov 03 02:36:24.328729 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.328617 4992 scope.go:122] "RemoveContainer" containerID="2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b" Nov 03 02:36:24.328907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.328607 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7" event={"ID":"af1bf105-6ac1-4d23-b5ce-904d17e86037","Type":"ContainerDied","Data":"b78f7c39ee996213f4c6653b4af78927d441e9db970637c0319caf4cd1f73c98"} Nov 03 02:36:24.337989 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.337955 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.338113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.337993 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh479\" (UniqueName: \"kubernetes.io/projected/af1bf105-6ac1-4d23-b5ce-904d17e86037-kube-api-access-gh479\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.338113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.338010 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.338113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.338024 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af1bf105-6ac1-4d23-b5ce-904d17e86037-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:36:24.339958 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.339791 4992 scope.go:122] "RemoveContainer" containerID="2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b" Nov 03 02:36:24.340201 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:36:24.340168 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b\": container with ID starting with 2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b not found: ID does not exist" containerID="2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b" Nov 03 02:36:24.340297 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.340207 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b"} err="failed to get container status \"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b\": rpc error: code = NotFound desc = could not find container \"2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b\": container with ID starting with 2aefe89486fbc9cfa8f4e9f635e5d43372bee05102d557293bc768d88833544b not found: ID does not exist" Nov 03 02:36:24.500830 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.500788 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 02:36:24.505339 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.505311 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-vrqw7"] Nov 03 02:36:24.604275 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:36:24.604239 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" path="/var/lib/kubelet/pods/af1bf105-6ac1-4d23-b5ce-904d17e86037/volumes" Nov 03 02:36:32.151956 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:36:32.151913 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:37:02.151982 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:37:02.151941 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:37:32.151776 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:37:32.151737 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:37:45.296486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:45.296460 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:37:45.301321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:45.301298 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:37:56.344082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.343994 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:37:56.344607 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.344227 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" containerID="cri-o://3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734" gracePeriod=30 Nov 03 02:37:56.395838 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.395796 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:37:56.396048 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.396026 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" containerID="cri-o://224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286" gracePeriod=30 Nov 03 02:37:56.398141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.398112 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:37:56.398299 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.398273 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" containerID="cri-o://bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8" gracePeriod=30 Nov 03 02:37:56.447800 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.447762 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:37:56.448003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:56.447982 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" containerID="cri-o://b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee" gracePeriod=30 Nov 03 02:37:59.411352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.411303 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-m8x8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:37:59.411876 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.411359 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:37:59.507380 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.507339 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4d5mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:37:59.507545 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.507398 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:37:59.635188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.635148 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-fqhg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:37:59.635392 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:37:59.635211 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:38:01.692690 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.692657 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:38:01.783217 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.783189 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:38:01.796315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.796286 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:38:01.801316 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.801288 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:38:01.827046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827014 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827066 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827089 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827113 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827157 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827213 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827276 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827380 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827378 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.827511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827401 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\") pod \"09c89df1-b20c-444a-ba37-65b8ce50eae0\" (UID: \"09c89df1-b20c-444a-ba37-65b8ce50eae0\") " Nov 03 02:38:01.827511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827495 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:01.827787 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827713 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.827787 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827743 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.827950 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.827923 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.828055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.828039 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.828112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.828062 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.828112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.828077 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c89df1-b20c-444a-ba37-65b8ce50eae0-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.828112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.828093 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.828112 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.828107 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.829767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.829739 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:01.829884 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.829803 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.830011 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.829991 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.830062 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.830046 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr" pod "09c89df1-b20c-444a-ba37-65b8ce50eae0" (UID: "09c89df1-b20c-444a-ba37-65b8ce50eae0"). InnerVolumeSpecName "kube-api-access-mwrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.925364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.925322 4992 generic.go:356] "Generic (PLEG): container finished" podID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerID="3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734" exitCode=0 Nov 03 02:38:01.925551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.925369 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" Nov 03 02:38:01.925551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.925410 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" event={"ID":"09c89df1-b20c-444a-ba37-65b8ce50eae0","Type":"ContainerDied","Data":"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734"} Nov 03 02:38:01.925551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.925459 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g" event={"ID":"09c89df1-b20c-444a-ba37-65b8ce50eae0","Type":"ContainerDied","Data":"d69b2bf0cee7d9c1a046a58a0acb0fc061fa67a483091328e95502362eb04293"} Nov 03 02:38:01.925551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.925488 4992 scope.go:122] "RemoveContainer" containerID="3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734" Nov 03 02:38:01.926861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.926838 4992 generic.go:356] "Generic (PLEG): container finished" podID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerID="b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee" exitCode=0 Nov 03 02:38:01.926970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.926884 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" Nov 03 02:38:01.926970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.926918 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" event={"ID":"094ee459-5323-4cd3-bf3e-db8e9a49f56c","Type":"ContainerDied","Data":"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee"} Nov 03 02:38:01.927083 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.926975 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq" event={"ID":"094ee459-5323-4cd3-bf3e-db8e9a49f56c","Type":"ContainerDied","Data":"ddf97f154a0d10b73e6228a28bcc80b7f20925d2898da2bd3cdfdaa7017198ea"} Nov 03 02:38:01.928396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928375 4992 generic.go:356] "Generic (PLEG): container finished" podID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerID="224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286" exitCode=0 Nov 03 02:38:01.928510 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928420 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" Nov 03 02:38:01.928510 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928454 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" event={"ID":"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26","Type":"ContainerDied","Data":"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286"} Nov 03 02:38:01.928510 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928489 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr" event={"ID":"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26","Type":"ContainerDied","Data":"db572fd39bedfc98126f1ad84b4e800bb17b747a986256298591611f06de8782"} Nov 03 02:38:01.928756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928433 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.928756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928684 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.928756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928704 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.928756 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928743 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.928925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928791 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.928925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928826 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.928925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928860 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.928925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928887 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.928925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928914 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.928985 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929015 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929042 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929164 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929076 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929175 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929224 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929250 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929283 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929310 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929341 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929388 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929428 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929462 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929536 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929513 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929549 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.929766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929653 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929739 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\") pod \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\" (UID: \"094ee459-5323-4cd3-bf3e-db8e9a49f56c\") " Nov 03 02:38:01.929867 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929777 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\") pod \"36371633-6ead-46ab-9bda-65014683a63a\" (UID: \"36371633-6ead-46ab-9bda-65014683a63a\") " Nov 03 02:38:01.929867 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.929811 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\") pod \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\" (UID: \"69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26\") " Nov 03 02:38:01.930383 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930365 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.930434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930390 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.930434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930406 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.930434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930421 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c89df1-b20c-444a-ba37-65b8ce50eae0-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.930586 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930436 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrtr\" (UniqueName: \"kubernetes.io/projected/09c89df1-b20c-444a-ba37-65b8ce50eae0-kube-api-access-mwrtr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:01.930938 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.930915 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.931072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931047 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.931159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931119 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.931284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931264 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:01.931601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931470 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.931601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931536 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.931739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.931690 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.932102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932077 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.932315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932294 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.932511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932491 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.932690 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932651 4992 generic.go:356] "Generic (PLEG): container finished" podID="36371633-6ead-46ab-9bda-65014683a63a" containerID="bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8" exitCode=0 Nov 03 02:38:01.932766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932738 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" event={"ID":"36371633-6ead-46ab-9bda-65014683a63a","Type":"ContainerDied","Data":"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8"} Nov 03 02:38:01.932828 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932778 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" event={"ID":"36371633-6ead-46ab-9bda-65014683a63a","Type":"ContainerDied","Data":"2fa0043d2474184ca35d3b5af0edb85512f06637b2c695c1d979efe216534e81"} Nov 03 02:38:01.932885 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932846 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2" Nov 03 02:38:01.933018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.932990 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:01.933142 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933120 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.933341 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933312 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.933432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933411 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:38:01.933660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933634 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.933919 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933896 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.934030 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.933996 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.934860 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.934831 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:38:01.937025 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.936961 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:01.937025 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.936987 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.937170 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.937119 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.937366 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.937334 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:01.937467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.937359 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:38:01.937614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.937595 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x" pod "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" (UID: "69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26"). InnerVolumeSpecName "kube-api-access-vtt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.939001 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.938981 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh" pod "094ee459-5323-4cd3-bf3e-db8e9a49f56c" (UID: "094ee459-5323-4cd3-bf3e-db8e9a49f56c"). InnerVolumeSpecName "kube-api-access-vk4jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.939080 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.939059 4992 scope.go:122] "RemoveContainer" containerID="3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734" Nov 03 02:38:01.939345 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:01.939324 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734\": container with ID starting with 3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734 not found: ID does not exist" containerID="3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734" Nov 03 02:38:01.939425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.939352 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734"} err="failed to get container status \"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734\": rpc error: code = NotFound desc = could not find container \"3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734\": container with ID starting with 3b689ec60f4741567fcfe39673120edfbcd7d856644f00242ecdcc1d0327e734 not found: ID does not exist" Nov 03 02:38:01.939425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.939371 4992 scope.go:122] "RemoveContainer" containerID="b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee" Nov 03 02:38:01.939884 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.939862 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t" pod "36371633-6ead-46ab-9bda-65014683a63a" (UID: "36371633-6ead-46ab-9bda-65014683a63a"). InnerVolumeSpecName "kube-api-access-7mw2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:38:01.948889 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.948867 4992 scope.go:122] "RemoveContainer" containerID="b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee" Nov 03 02:38:01.949269 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:01.949247 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee\": container with ID starting with b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee not found: ID does not exist" containerID="b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee" Nov 03 02:38:01.949333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.949277 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee"} err="failed to get container status \"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee\": rpc error: code = NotFound desc = could not find container \"b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee\": container with ID starting with b196de9c325ea1e261865c23e9c5dc4e6268d08093f11796325fb6e627b7e0ee not found: ID does not exist" Nov 03 02:38:01.949333 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.949296 4992 scope.go:122] "RemoveContainer" containerID="224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286" Nov 03 02:38:01.959071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.959051 4992 scope.go:122] "RemoveContainer" containerID="224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286" Nov 03 02:38:01.959363 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:01.959343 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286\": container with ID starting with 224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286 not found: ID does not exist" containerID="224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286" Nov 03 02:38:01.959409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.959378 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286"} err="failed to get container status \"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286\": rpc error: code = NotFound desc = could not find container \"224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286\": container with ID starting with 224b7a1f4dc7e372c6c3c8776c2557d1d90892698adf5eaca5a48c4464a89286 not found: ID does not exist" Nov 03 02:38:01.959409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.959397 4992 scope.go:122] "RemoveContainer" containerID="bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8" Nov 03 02:38:01.968717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.968695 4992 scope.go:122] "RemoveContainer" containerID="bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8" Nov 03 02:38:01.969050 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:01.968994 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8\": container with ID starting with bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8 not found: ID does not exist" containerID="bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8" Nov 03 02:38:01.969118 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:01.969060 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8"} err="failed to get container status \"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8\": rpc error: code = NotFound desc = could not find container \"bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8\": container with ID starting with bebe7865442d5fd089904d8b4fe5245230cb68af1f3b1e71a9e6b692819578c8 not found: ID does not exist" Nov 03 02:38:02.032059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032021 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032054 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032059 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032063 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032074 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36371633-6ead-46ab-9bda-65014683a63a-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032082 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032093 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032100 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032108 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032116 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032123 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032132 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtt8x\" (UniqueName: \"kubernetes.io/projected/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-kube-api-access-vtt8x\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032141 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032149 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4jh\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-kube-api-access-vk4jh\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032157 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36371633-6ead-46ab-9bda-65014683a63a-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032167 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032174 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032184 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mw2t\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-kube-api-access-7mw2t\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032193 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032201 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032208 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032216 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032224 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032234 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032241 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094ee459-5323-4cd3-bf3e-db8e9a49f56c-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032250 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36371633-6ead-46ab-9bda-65014683a63a-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.032293 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.032259 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36371633-6ead-46ab-9bda-65014683a63a-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:38:02.165661 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:02.165619 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:38:02.352542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.352502 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:38:02.357983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.357949 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-m8x8g"] Nov 03 02:38:02.599214 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.599116 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" path="/var/lib/kubelet/pods/09c89df1-b20c-444a-ba37-65b8ce50eae0/volumes" Nov 03 02:38:02.635663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.635625 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:38:02.640120 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.640083 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-fqhg2"] Nov 03 02:38:02.753021 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.752986 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:38:02.757821 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.757784 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bccqr"] Nov 03 02:38:02.871968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.871877 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:38:02.876826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:02.876794 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4d5mq"] Nov 03 02:38:04.594824 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:04.594791 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" path="/var/lib/kubelet/pods/094ee459-5323-4cd3-bf3e-db8e9a49f56c/volumes" Nov 03 02:38:04.595322 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:04.595260 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36371633-6ead-46ab-9bda-65014683a63a" path="/var/lib/kubelet/pods/36371633-6ead-46ab-9bda-65014683a63a/volumes" Nov 03 02:38:04.595677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:38:04.595664 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" path="/var/lib/kubelet/pods/69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26/volumes" Nov 03 02:38:32.151863 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:38:32.151822 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:39:02.152620 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:39:02.152576 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:39:30.783934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:30.783899 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:39:32.151757 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:39:32.151716 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:39:33.807301 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:33.807259 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:39:33.807843 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:33.807574 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" podUID="dd1594ec-aadb-4907-8b86-a1c1894d2969" containerName="maas-api" containerID="cri-o://2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c" gracePeriod=30 Nov 03 02:39:34.138766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.138734 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:39:34.197415 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.197378 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\") pod \"dd1594ec-aadb-4907-8b86-a1c1894d2969\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " Nov 03 02:39:34.197677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.197431 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\") pod \"dd1594ec-aadb-4907-8b86-a1c1894d2969\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " Nov 03 02:39:34.197677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.197467 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\") pod \"dd1594ec-aadb-4907-8b86-a1c1894d2969\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " Nov 03 02:39:34.197926 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.197896 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca" pod "dd1594ec-aadb-4907-8b86-a1c1894d2969" (UID: "dd1594ec-aadb-4907-8b86-a1c1894d2969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:39:34.198013 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.197960 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\" (UniqueName: \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\") pod \"dd1594ec-aadb-4907-8b86-a1c1894d2969\" (UID: \"dd1594ec-aadb-4907-8b86-a1c1894d2969\") " Nov 03 02:39:34.198423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.198399 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd1594ec-aadb-4907-8b86-a1c1894d2969-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:34.200234 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.200205 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls" pod "dd1594ec-aadb-4907-8b86-a1c1894d2969" (UID: "dd1594ec-aadb-4907-8b86-a1c1894d2969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:39:34.200500 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.200469 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q" pod "dd1594ec-aadb-4907-8b86-a1c1894d2969" (UID: "dd1594ec-aadb-4907-8b86-a1c1894d2969"). InnerVolumeSpecName "kube-api-access-h7p6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:34.200870 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.200849 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs" pod "dd1594ec-aadb-4907-8b86-a1c1894d2969" (UID: "dd1594ec-aadb-4907-8b86-a1c1894d2969"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:39:34.299130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.299089 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:34.299130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.299122 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd1594ec-aadb-4907-8b86-a1c1894d2969-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:34.299130 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.299132 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7p6q\" (UniqueName: \"kubernetes.io/projected/dd1594ec-aadb-4907-8b86-a1c1894d2969-kube-api-access-h7p6q\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:34.487684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.487642 4992 generic.go:356] "Generic (PLEG): container finished" podID="dd1594ec-aadb-4907-8b86-a1c1894d2969" containerID="2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c" exitCode=0 Nov 03 02:39:34.487897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.487691 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" Nov 03 02:39:34.487897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.487738 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" event={"ID":"dd1594ec-aadb-4907-8b86-a1c1894d2969","Type":"ContainerDied","Data":"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c"} Nov 03 02:39:34.487897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.487799 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z" event={"ID":"dd1594ec-aadb-4907-8b86-a1c1894d2969","Type":"ContainerDied","Data":"14034736769eb56e5794bb8fc76f0973536fc179a638f5bfca2cac4ff9a3c216"} Nov 03 02:39:34.487897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.487828 4992 scope.go:122] "RemoveContainer" containerID="2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c" Nov 03 02:39:34.499446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.499172 4992 scope.go:122] "RemoveContainer" containerID="2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c" Nov 03 02:39:34.501229 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:39:34.501198 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c\": container with ID starting with 2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c not found: ID does not exist" containerID="2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c" Nov 03 02:39:34.501337 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.501232 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c"} err="failed to get container status \"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c\": rpc error: code = NotFound desc = could not find container \"2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c\": container with ID starting with 2f169327df92af6d4a96f50d4fed94c8c27b4a9c773d46df832193730e8d1c7c not found: ID does not exist" Nov 03 02:39:34.660032 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.659992 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:39:34.713582 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:34.712826 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c34879-7d9cf8f4cb-tz64z"] Nov 03 02:39:36.595142 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:36.595106 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1594ec-aadb-4907-8b86-a1c1894d2969" path="/var/lib/kubelet/pods/dd1594ec-aadb-4907-8b86-a1c1894d2969/volumes" Nov 03 02:39:45.804448 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:45.804403 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerName="main" containerID="cri-o://fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a" gracePeriod=45 Nov 03 02:39:45.808693 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:45.808652 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": EOF" Nov 03 02:39:46.079148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.079119 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:39:46.128995 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.128958 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129010 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\" (UniqueName: \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129045 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129065 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129105 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129179 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\") pod \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\" (UID: \"a66dc16f-e1a9-4462-845d-3c5b35e86ce7\") " Nov 03 02:39:46.129456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129341 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:46.129456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129354 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:46.129456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129385 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:46.129717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129616 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.129717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129633 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.129717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.129643 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.131678 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.131650 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:39:46.131856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.131831 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:39:46.131916 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.131853 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q" pod "a66dc16f-e1a9-4462-845d-3c5b35e86ce7" (UID: "a66dc16f-e1a9-4462-845d-3c5b35e86ce7"). InnerVolumeSpecName "kube-api-access-86p8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:39:46.230638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.230594 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86p8q\" (UniqueName: \"kubernetes.io/projected/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-kube-api-access-86p8q\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.230638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.230632 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.230638 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.230646 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a66dc16f-e1a9-4462-845d-3c5b35e86ce7-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:39:46.559169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.559132 4992 generic.go:356] "Generic (PLEG): container finished" podID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerID="fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a" exitCode=0 Nov 03 02:39:46.559360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.559183 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" Nov 03 02:39:46.559360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.559224 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" event={"ID":"a66dc16f-e1a9-4462-845d-3c5b35e86ce7","Type":"ContainerDied","Data":"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a"} Nov 03 02:39:46.559360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.559274 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw" event={"ID":"a66dc16f-e1a9-4462-845d-3c5b35e86ce7","Type":"ContainerDied","Data":"5aeb9f7f64c67321b0a1da044f4c53fa45dc4b15520a32dc1b832152fdf2eeb6"} Nov 03 02:39:46.559360 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.559294 4992 scope.go:122] "RemoveContainer" containerID="fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a" Nov 03 02:39:46.569428 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.569406 4992 scope.go:122] "RemoveContainer" containerID="fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a" Nov 03 02:39:46.569807 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:39:46.569780 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a\": container with ID starting with fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a not found: ID does not exist" containerID="fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a" Nov 03 02:39:46.569917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.569815 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a"} err="failed to get container status \"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a\": rpc error: code = NotFound desc = could not find container \"fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a\": container with ID starting with fff81ceefac6076a697534517182d6ef2b18b1284885247b3b3968e082a0377a not found: ID does not exist" Nov 03 02:39:46.729517 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.729478 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:39:46.733734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:46.733707 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/rate-test-model-6a5d73-kserve-55f958589-mxvnw"] Nov 03 02:39:48.594632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:39:48.594595 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" path="/var/lib/kubelet/pods/a66dc16f-e1a9-4462-845d-3c5b35e86ce7/volumes" Nov 03 02:40:02.151249 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:02.151211 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:40:12.116757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:12.116716 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 02:40:12.167100 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:12.167062 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 02:40:27.138389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.138342 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" podUID="9d9b1098-bb8d-4333-9c05-5a206aacb738" containerName="main" containerID="cri-o://6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae" gracePeriod=45 Nov 03 02:40:27.187808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.187763 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" podUID="9ca40087-ecfa-4848-bafc-71ec434da390" containerName="main" containerID="cri-o://7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911" gracePeriod=45 Nov 03 02:40:27.493175 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.493148 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 02:40:27.506734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.506700 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 02:40:27.585665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585622 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.585881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585725 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.585881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585788 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.585881 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585831 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.586070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585917 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585955 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\" (UniqueName: \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.585982 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.586011 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.586047 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.586073 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.586352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.586153 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\") pod \"9d9b1098-bb8d-4333-9c05-5a206aacb738\" (UID: \"9d9b1098-bb8d-4333-9c05-5a206aacb738\") " Nov 03 02:40:27.586352 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.586203 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\" (UniqueName: \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\") pod \"9ca40087-ecfa-4848-bafc-71ec434da390\" (UID: \"9ca40087-ecfa-4848-bafc-71ec434da390\") " Nov 03 02:40:27.587924 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.587893 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.588278 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.588248 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.588617 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.588591 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.588862 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.588835 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.589642 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.589298 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.589854 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.589815 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "kube-api-access-9dbk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:27.590122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.589992 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.590122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.590076 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "kube-api-access-8khht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:27.590846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.590820 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.591548 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.591525 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs" pod "9d9b1098-bb8d-4333-9c05-5a206aacb738" (UID: "9d9b1098-bb8d-4333-9c05-5a206aacb738"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:27.591880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.591858 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:40:27.592212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.592189 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm" pod "9ca40087-ecfa-4848-bafc-71ec434da390" (UID: "9ca40087-ecfa-4848-bafc-71ec434da390"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:27.688416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688377 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688416 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca40087-ecfa-4848-bafc-71ec434da390-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688416 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688432 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688449 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688465 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688480 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dbk4\" (UniqueName: \"kubernetes.io/projected/9d9b1098-bb8d-4333-9c05-5a206aacb738-kube-api-access-9dbk4\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688495 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688508 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688530 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9b1098-bb8d-4333-9c05-5a206aacb738-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688545 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca40087-ecfa-4848-bafc-71ec434da390-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688554 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9b1098-bb8d-4333-9c05-5a206aacb738-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.688767 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.688584 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khht\" (UniqueName: \"kubernetes.io/projected/9ca40087-ecfa-4848-bafc-71ec434da390-kube-api-access-8khht\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:27.800537 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.800503 4992 generic.go:356] "Generic (PLEG): container finished" podID="9d9b1098-bb8d-4333-9c05-5a206aacb738" containerID="6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae" exitCode=0 Nov 03 02:40:27.800733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.800585 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" Nov 03 02:40:27.800733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.800602 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" event={"ID":"9d9b1098-bb8d-4333-9c05-5a206aacb738","Type":"ContainerDied","Data":"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae"} Nov 03 02:40:27.800733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.800643 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c" event={"ID":"9d9b1098-bb8d-4333-9c05-5a206aacb738","Type":"ContainerDied","Data":"2213669c2a86996704c652b3877c4a262d974e759e3f5ece844d9669a3fb6482"} Nov 03 02:40:27.800733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.800659 4992 scope.go:122] "RemoveContainer" containerID="6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae" Nov 03 02:40:27.802154 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.802131 4992 generic.go:356] "Generic (PLEG): container finished" podID="9ca40087-ecfa-4848-bafc-71ec434da390" containerID="7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911" exitCode=0 Nov 03 02:40:27.802273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.802181 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" Nov 03 02:40:27.802273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.802183 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" event={"ID":"9ca40087-ecfa-4848-bafc-71ec434da390","Type":"ContainerDied","Data":"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911"} Nov 03 02:40:27.802273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.802225 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7" event={"ID":"9ca40087-ecfa-4848-bafc-71ec434da390","Type":"ContainerDied","Data":"84b6a776521b7ce4bb855e83a21dc9fc91dd12e2ceca374eb0c01280a305780f"} Nov 03 02:40:27.811210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.811190 4992 scope.go:122] "RemoveContainer" containerID="6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae" Nov 03 02:40:27.811727 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:27.811575 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae\": container with ID starting with 6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae not found: ID does not exist" containerID="6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae" Nov 03 02:40:27.811727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.811617 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae"} err="failed to get container status \"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae\": rpc error: code = NotFound desc = could not find container \"6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae\": container with ID starting with 6b1316e417a5581adabbc9dacc62a4fc0b70b83305b2881a48eb6f7882f2d2ae not found: ID does not exist" Nov 03 02:40:27.811727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.811641 4992 scope.go:122] "RemoveContainer" containerID="7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911" Nov 03 02:40:27.822226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.822195 4992 scope.go:122] "RemoveContainer" containerID="7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911" Nov 03 02:40:27.822686 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:27.822656 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911\": container with ID starting with 7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911 not found: ID does not exist" containerID="7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911" Nov 03 02:40:27.822778 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.822720 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911"} err="failed to get container status \"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911\": rpc error: code = NotFound desc = could not find container \"7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911\": container with ID starting with 7d47f14bbaae2d4fb531cc219847bcfdf75b27182348e0fed978ed3d718be911 not found: ID does not exist" Nov 03 02:40:27.970735 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.970642 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 02:40:27.976259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:27.976225 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-w3-41aa5f-kserve-b85867c76-n6bj7"] Nov 03 02:40:28.088217 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:28.088174 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 02:40:28.092099 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:28.092068 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41aa5f/distinct-2-w3-41aa5f-kserve-5c8578488c-kdk7c"] Nov 03 02:40:28.595158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:28.595126 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca40087-ecfa-4848-bafc-71ec434da390" path="/var/lib/kubelet/pods/9ca40087-ecfa-4848-bafc-71ec434da390/volumes" Nov 03 02:40:28.595600 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:28.595583 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9b1098-bb8d-4333-9c05-5a206aacb738" path="/var/lib/kubelet/pods/9d9b1098-bb8d-4333-9c05-5a206aacb738/volumes" Nov 03 02:40:32.151576 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:32.151527 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:40:39.204902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:39.204857 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:40:39.205408 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:39.205135 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" containerID="cri-o://c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497" gracePeriod=30 Nov 03 02:40:44.474748 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.474676 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:40:44.645539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.645505 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 02:40:44.676268 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676215 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676308 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676344 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676390 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676492 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676460 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676518 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676604 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676645 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.676973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.676765 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\") pod \"acff4567-6d9a-40f4-9349-ac43f440dba4\" (UID: \"acff4567-6d9a-40f4-9349-ac43f440dba4\") " Nov 03 02:40:44.677153 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.677024 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:44.677273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.677217 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:44.677273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.677129 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:40:44.677403 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.677337 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:44.678073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.678045 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.678180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.678080 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.678180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.678097 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.678180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.678119 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acff4567-6d9a-40f4-9349-ac43f440dba4-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.678361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.678261 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:44.679845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.679817 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:40:44.680742 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.680715 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:40:44.680852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.680732 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "kube-api-access-hrq4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:44.692584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.692535 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token" pod "acff4567-6d9a-40f4-9349-ac43f440dba4" (UID: "acff4567-6d9a-40f4-9349-ac43f440dba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:40:44.699662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.699632 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 02:40:44.778734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.778648 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrq4s\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-kube-api-access-hrq4s\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.778734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.778677 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.778734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.778687 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.778734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.778697 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.778734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.778706 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acff4567-6d9a-40f4-9349-ac43f440dba4-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:40:44.904836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.901909 4992 generic.go:356] "Generic (PLEG): container finished" podID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerID="c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497" exitCode=0 Nov 03 02:40:44.904836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.901993 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerDied","Data":"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497"} Nov 03 02:40:44.904836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.902069 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" event={"ID":"acff4567-6d9a-40f4-9349-ac43f440dba4","Type":"ContainerDied","Data":"c07e16e864ac4f211cb727a21246e932ce5167f9938ab813da12adbcb054fb5d"} Nov 03 02:40:44.904836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.902000 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t" Nov 03 02:40:44.904836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.902098 4992 scope.go:122] "RemoveContainer" containerID="c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497" Nov 03 02:40:44.914231 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.914208 4992 scope.go:122] "RemoveContainer" containerID="4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d" Nov 03 02:40:44.923695 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.923669 4992 scope.go:122] "RemoveContainer" containerID="c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497" Nov 03 02:40:44.924028 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:44.924007 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497\": container with ID starting with c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497 not found: ID does not exist" containerID="c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497" Nov 03 02:40:44.924113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.924043 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497"} err="failed to get container status \"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497\": rpc error: code = NotFound desc = could not find container \"c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497\": container with ID starting with c3f494b361780e89ca7b03b7824de902667c2ccb5af1a0ef173299281fe33497 not found: ID does not exist" Nov 03 02:40:44.924113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.924070 4992 scope.go:122] "RemoveContainer" containerID="4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d" Nov 03 02:40:44.924360 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:40:44.924342 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d\": container with ID starting with 4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d not found: ID does not exist" containerID="4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d" Nov 03 02:40:44.924402 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:44.924370 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d"} err="failed to get container status \"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d\": rpc error: code = NotFound desc = could not find container \"4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d\": container with ID starting with 4552e8b4e65a0cd6c461fea8c00f33ae6bcdc943069087c89e12dd325613162d not found: ID does not exist" Nov 03 02:40:45.071717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:45.071631 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:40:45.076029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:45.076000 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9576a-openshift-default-b46f8977c-jnz7t"] Nov 03 02:40:46.594777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:46.594745 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" path="/var/lib/kubelet/pods/acff4567-6d9a-40f4-9349-ac43f440dba4/volumes" Nov 03 02:40:59.666173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.666087 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" containerID="cri-o://f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d" gracePeriod=45 Nov 03 02:40:59.720232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.720188 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" containerID="cri-o://009b3d54dfba0651e51e8b7a41736a390670913657ce805f27312c8ab2da5456" gracePeriod=45 Nov 03 02:40:59.861772 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.861727 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Nov 03 02:40:59.983535 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.983502 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 02:40:59.992258 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.992224 4992 generic.go:356] "Generic (PLEG): container finished" podID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerID="009b3d54dfba0651e51e8b7a41736a390670913657ce805f27312c8ab2da5456" exitCode=0 Nov 03 02:40:59.992425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.992311 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" event={"ID":"783a8dcd-1583-4cbe-9b69-c4a970b09d56","Type":"ContainerDied","Data":"009b3d54dfba0651e51e8b7a41736a390670913657ce805f27312c8ab2da5456"} Nov 03 02:40:59.994097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.994074 4992 generic.go:356] "Generic (PLEG): container finished" podID="210111a3-f7e5-4034-9011-3654ab84ff02" containerID="f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d" exitCode=0 Nov 03 02:40:59.994238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.994123 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" Nov 03 02:40:59.994238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.994157 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" event={"ID":"210111a3-f7e5-4034-9011-3654ab84ff02","Type":"ContainerDied","Data":"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d"} Nov 03 02:40:59.994238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.994195 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" event={"ID":"210111a3-f7e5-4034-9011-3654ab84ff02","Type":"ContainerDied","Data":"b2225696338756e324d918fa69887a54d326ffbe08dc5692333f026607008991"} Nov 03 02:40:59.994238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:40:59.994213 4992 scope.go:122] "RemoveContainer" containerID="f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d" Nov 03 02:41:00.013394 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.013373 4992 scope.go:122] "RemoveContainer" containerID="f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d" Nov 03 02:41:00.013746 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:00.013722 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d\": container with ID starting with f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d not found: ID does not exist" containerID="f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d" Nov 03 02:41:00.013958 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.013758 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d"} err="failed to get container status \"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d\": rpc error: code = NotFound desc = could not find container \"f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d\": container with ID starting with f9a199430e88443d718931e84901956694ae8a42f95a58a95dd3fd265e4a080d not found: ID does not exist" Nov 03 02:41:00.023224 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.023194 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 02:41:00.045124 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045094 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045124 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045132 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045204 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\" (UniqueName: \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045248 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045292 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045362 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\" (UniqueName: \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\") pod \"210111a3-f7e5-4034-9011-3654ab84ff02\" (UID: \"210111a3-f7e5-4034-9011-3654ab84ff02\") " Nov 03 02:41:00.045616 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045431 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.045616 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045445 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.045616 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045604 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.045807 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045793 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.045856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045809 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.045856 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.045819 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.047663 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.047636 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.047765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.047671 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "kube-api-access-k59xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:00.047765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.047744 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs" pod "210111a3-f7e5-4034-9011-3654ab84ff02" (UID: "210111a3-f7e5-4034-9011-3654ab84ff02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:00.146646 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146609 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.146848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146663 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.146848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146704 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\" (UniqueName: \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.146848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146797 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.146848 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146830 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\" (UniqueName: \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.147079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.146926 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\") pod \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\" (UID: \"783a8dcd-1583-4cbe-9b69-c4a970b09d56\") " Nov 03 02:41:00.147267 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147218 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.147379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147277 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/210111a3-f7e5-4034-9011-3654ab84ff02-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.147445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147404 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/210111a3-f7e5-4034-9011-3654ab84ff02-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.147445 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147437 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k59xw\" (UniqueName: \"kubernetes.io/projected/210111a3-f7e5-4034-9011-3654ab84ff02-kube-api-access-k59xw\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.147526 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147311 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.147526 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.147361 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.149119 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.149097 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "kube-api-access-m48r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:00.149554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.149532 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:00.149554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.149542 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs" pod "783a8dcd-1583-4cbe-9b69-c4a970b09d56" (UID: "783a8dcd-1583-4cbe-9b69-c4a970b09d56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248144 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248178 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248192 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248205 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783a8dcd-1583-4cbe-9b69-c4a970b09d56-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248216 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783a8dcd-1583-4cbe-9b69-c4a970b09d56-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.248238 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.248232 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m48r5\" (UniqueName: \"kubernetes.io/projected/783a8dcd-1583-4cbe-9b69-c4a970b09d56-kube-api-access-m48r5\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:00.418315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.418276 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 02:41:00.422024 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.421994 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf"] Nov 03 02:41:00.595250 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.595167 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" path="/var/lib/kubelet/pods/210111a3-f7e5-4034-9011-3654ab84ff02/volumes" Nov 03 02:41:00.901797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:00.901704 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-w4-0bf34a-kserve-d57964647-mxwkf" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 02:41:01.001748 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:01.001694 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" Nov 03 02:41:01.001968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:01.001693 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2" event={"ID":"783a8dcd-1583-4cbe-9b69-c4a970b09d56","Type":"ContainerDied","Data":"7447f38a5ba659d5f9cc7366af6e6e4553cfbffc6541779dd9cddca2b68e73bd"} Nov 03 02:41:01.001968 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:01.001833 4992 scope.go:122] "RemoveContainer" containerID="009b3d54dfba0651e51e8b7a41736a390670913657ce805f27312c8ab2da5456" Nov 03 02:41:01.171220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:01.171180 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 02:41:01.222970 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:01.222926 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/premium-w4-0bf34a-kserve-5cdcb99bd8-dbkt2"] Nov 03 02:41:02.150730 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:02.150694 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:41:02.594884 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:02.594850 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" path="/var/lib/kubelet/pods/783a8dcd-1583-4cbe-9b69-c4a970b09d56/volumes" Nov 03 02:41:03.343088 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.343051 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 02:41:03.343617 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.343271 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" podUID="9703362e-f746-4a90-8dba-e1fe1368d960" containerName="maas-api" containerID="cri-o://12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566" gracePeriod=30 Nov 03 02:41:03.615769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.615741 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 02:41:03.687236 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.687193 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\" (UniqueName: \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\") pod \"9703362e-f746-4a90-8dba-e1fe1368d960\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " Nov 03 02:41:03.687391 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.687311 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\") pod \"9703362e-f746-4a90-8dba-e1fe1368d960\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " Nov 03 02:41:03.687391 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.687386 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\") pod \"9703362e-f746-4a90-8dba-e1fe1368d960\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " Nov 03 02:41:03.687495 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.687472 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") pod \"9703362e-f746-4a90-8dba-e1fe1368d960\" (UID: \"9703362e-f746-4a90-8dba-e1fe1368d960\") " Nov 03 02:41:03.687976 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.687846 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca" pod "9703362e-f746-4a90-8dba-e1fe1368d960" (UID: "9703362e-f746-4a90-8dba-e1fe1368d960"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:03.689840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.689808 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls" pod "9703362e-f746-4a90-8dba-e1fe1368d960" (UID: "9703362e-f746-4a90-8dba-e1fe1368d960"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:03.690016 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.689996 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs" pod "9703362e-f746-4a90-8dba-e1fe1368d960" (UID: "9703362e-f746-4a90-8dba-e1fe1368d960"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:03.690066 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.690018 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm" pod "9703362e-f746-4a90-8dba-e1fe1368d960" (UID: "9703362e-f746-4a90-8dba-e1fe1368d960"). InnerVolumeSpecName "kube-api-access-d25tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:03.788361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.788323 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:03.788361 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.788361 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d25tm\" (UniqueName: \"kubernetes.io/projected/9703362e-f746-4a90-8dba-e1fe1368d960-kube-api-access-d25tm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:03.788610 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.788378 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9703362e-f746-4a90-8dba-e1fe1368d960-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:03.788610 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:03.788390 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9703362e-f746-4a90-8dba-e1fe1368d960-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:04.020131 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.020096 4992 generic.go:356] "Generic (PLEG): container finished" podID="9703362e-f746-4a90-8dba-e1fe1368d960" containerID="12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566" exitCode=0 Nov 03 02:41:04.020313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.020149 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" Nov 03 02:41:04.020313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.020182 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" event={"ID":"9703362e-f746-4a90-8dba-e1fe1368d960","Type":"ContainerDied","Data":"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566"} Nov 03 02:41:04.020313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.020241 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz" event={"ID":"9703362e-f746-4a90-8dba-e1fe1368d960","Type":"ContainerDied","Data":"9092e23e9929d1f079cbfe01307fa902db13d98c174fe0e57ef6511083e58edd"} Nov 03 02:41:04.020313 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.020265 4992 scope.go:122] "RemoveContainer" containerID="12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566" Nov 03 02:41:04.030486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.030468 4992 scope.go:122] "RemoveContainer" containerID="12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566" Nov 03 02:41:04.030770 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:04.030751 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566\": container with ID starting with 12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566 not found: ID does not exist" containerID="12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566" Nov 03 02:41:04.030817 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.030779 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566"} err="failed to get container status \"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566\": rpc error: code = NotFound desc = could not find container \"12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566\": container with ID starting with 12a15de05c6cc3861459e1e31bcbe243e300fe06607c20421b8b17a96b7b9566 not found: ID does not exist" Nov 03 02:41:04.189624 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.189585 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 02:41:04.241533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.241497 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-41aa5f-59674f6698-2kgqz"] Nov 03 02:41:04.595495 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:04.595457 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9703362e-f746-4a90-8dba-e1fe1368d960" path="/var/lib/kubelet/pods/9703362e-f746-4a90-8dba-e1fe1368d960/volumes" Nov 03 02:41:10.436918 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.436879 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:41:10.437417 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.437094 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" podUID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" containerName="maas-api" containerID="cri-o://915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61" gracePeriod=30 Nov 03 02:41:10.708508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.708476 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:41:10.864715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.864664 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\") pod \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " Nov 03 02:41:10.864898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.864747 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\" (UniqueName: \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\") pod \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " Nov 03 02:41:10.864898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.864840 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") pod \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " Nov 03 02:41:10.864898 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.864870 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\") pod \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\" (UID: \"e9fab307-193c-420b-bdcd-7a5e2c6d104e\") " Nov 03 02:41:10.865513 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.865479 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca" pod "e9fab307-193c-420b-bdcd-7a5e2c6d104e" (UID: "e9fab307-193c-420b-bdcd-7a5e2c6d104e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:10.868592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.867847 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6" pod "e9fab307-193c-420b-bdcd-7a5e2c6d104e" (UID: "e9fab307-193c-420b-bdcd-7a5e2c6d104e"). InnerVolumeSpecName "kube-api-access-bfqn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:10.868592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.867996 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs" pod "e9fab307-193c-420b-bdcd-7a5e2c6d104e" (UID: "e9fab307-193c-420b-bdcd-7a5e2c6d104e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:10.868733 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.868659 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls" pod "e9fab307-193c-420b-bdcd-7a5e2c6d104e" (UID: "e9fab307-193c-420b-bdcd-7a5e2c6d104e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:10.966029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.965944 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:10.966029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.965979 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9fab307-193c-420b-bdcd-7a5e2c6d104e-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:10.966029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.965989 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9fab307-193c-420b-bdcd-7a5e2c6d104e-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:10.966029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:10.965998 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqn6\" (UniqueName: \"kubernetes.io/projected/e9fab307-193c-420b-bdcd-7a5e2c6d104e-kube-api-access-bfqn6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:11.071768 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.071735 4992 generic.go:356] "Generic (PLEG): container finished" podID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" containerID="915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61" exitCode=0 Nov 03 02:41:11.071955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.071781 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" Nov 03 02:41:11.071955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.071810 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" event={"ID":"e9fab307-193c-420b-bdcd-7a5e2c6d104e","Type":"ContainerDied","Data":"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61"} Nov 03 02:41:11.071955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.071861 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v" event={"ID":"e9fab307-193c-420b-bdcd-7a5e2c6d104e","Type":"ContainerDied","Data":"63691a628aa61771460300a37e355d706ddc77c545d92ee87359e9c7af38bb83"} Nov 03 02:41:11.071955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.071887 4992 scope.go:122] "RemoveContainer" containerID="915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61" Nov 03 02:41:11.082531 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.082512 4992 scope.go:122] "RemoveContainer" containerID="915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61" Nov 03 02:41:11.082879 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:11.082860 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61\": container with ID starting with 915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61 not found: ID does not exist" containerID="915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61" Nov 03 02:41:11.082936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.082889 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61"} err="failed to get container status \"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61\": rpc error: code = NotFound desc = could not find container \"915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61\": container with ID starting with 915a2626957234755ef71a87e143604ccfd9d6d541b74a3656df40366d155d61 not found: ID does not exist" Nov 03 02:41:11.241930 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.241841 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:41:11.246022 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:11.245990 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6a5d73-74984dcc9f-gl24v"] Nov 03 02:41:12.594381 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:12.594345 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" path="/var/lib/kubelet/pods/e9fab307-193c-420b-bdcd-7a5e2c6d104e/volumes" Nov 03 02:41:15.810728 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:15.810690 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:41:15.811288 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:15.810923 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" containerID="cri-o://7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b" gracePeriod=30 Nov 03 02:41:16.502165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.502128 4992 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:41:16.502346 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.502197 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:41:16.862518 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862432 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72"] Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862885 4992 cpu_manager.go:414] "containerMap: removing container" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862899 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862911 4992 cpu_manager.go:414] "containerMap: removing container" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862916 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862924 4992 cpu_manager.go:414] "containerMap: removing container" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" Nov 03 02:41:16.862934 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862929 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862939 4992 cpu_manager.go:414] "containerMap: removing container" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862946 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862954 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9703362e-f746-4a90-8dba-e1fe1368d960" containerName="maas-api" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862958 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9703362e-f746-4a90-8dba-e1fe1368d960" containerName="maas-api" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862964 4992 cpu_manager.go:414] "containerMap: removing container" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862969 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862983 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9ca40087-ecfa-4848-bafc-71ec434da390" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862987 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca40087-ecfa-4848-bafc-71ec434da390" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862993 4992 cpu_manager.go:414] "containerMap: removing container" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.862998 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863005 4992 cpu_manager.go:414] "containerMap: removing container" podUID="9d9b1098-bb8d-4333-9c05-5a206aacb738" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863009 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9b1098-bb8d-4333-9c05-5a206aacb738" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863014 4992 cpu_manager.go:414] "containerMap: removing container" podUID="dd1594ec-aadb-4907-8b86-a1c1894d2969" containerName="maas-api" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863018 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1594ec-aadb-4907-8b86-a1c1894d2969" containerName="maas-api" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863030 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863034 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863040 4992 cpu_manager.go:414] "containerMap: removing container" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863044 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863060 4992 cpu_manager.go:414] "containerMap: removing container" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863065 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863080 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" containerName="maas-api" Nov 03 02:41:16.863133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863085 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" containerName="maas-api" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863149 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="783a8dcd-1583-4cbe-9b69-c4a970b09d56" containerName="main" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863159 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="36371633-6ead-46ab-9bda-65014683a63a" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863166 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c89df1-b20c-444a-ba37-65b8ce50eae0" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863171 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9703362e-f746-4a90-8dba-e1fe1368d960" containerName="maas-api" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863176 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1bf105-6ac1-4d23-b5ce-904d17e86037" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863182 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca40087-ecfa-4848-bafc-71ec434da390" containerName="main" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863187 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13cbd9f-652a-4eb3-872f-382758aca6b5" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863192 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ba8a9d-c4e3-4879-8cdd-f0cf4a6fdb26" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863198 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9b1098-bb8d-4333-9c05-5a206aacb738" containerName="main" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863202 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1594ec-aadb-4907-8b86-a1c1894d2969" containerName="maas-api" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863209 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66dc16f-e1a9-4462-845d-3c5b35e86ce7" containerName="main" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863214 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="210111a3-f7e5-4034-9011-3654ab84ff02" containerName="main" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863221 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="094ee459-5323-4cd3-bf3e-db8e9a49f56c" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863227 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863234 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fab307-193c-420b-bdcd-7a5e2c6d104e" containerName="maas-api" Nov 03 02:41:16.863851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.863240 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:16.868008 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.867989 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:16.921297 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.921256 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72"] Nov 03 02:41:16.922237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.922211 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:41:16.922407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.922337 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:41:16.926935 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.926903 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:16.927079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.926954 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:16.927079 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:16.926972 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7l5\" (UniqueName: \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.027990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.027947 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.028173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.028029 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.028173 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.028056 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7l5\" (UniqueName: \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.028780 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.028749 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.030462 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.030437 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.104651 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.104599 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7l5\" (UniqueName: \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\") pod \"maas-api-revoke-sub-8273e395-whv72\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.181760 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.181721 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:17.562932 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:41:17.562896 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc71c7f_ad8d_4296_9cce_6d304a9e18fe.slice/crio-4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa WatchSource:0}: Error finding container 4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa: Status 404 returned error can't find the container with id 4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa Nov 03 02:41:17.564147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:17.564119 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72"] Nov 03 02:41:18.123026 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:18.122936 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" event={"ID":"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe","Type":"ContainerStarted","Data":"e9d48044f66013992c2159f6dd57883e4c5d874a254f10614a81902840ab7ec9"} Nov 03 02:41:18.123026 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:18.122991 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" event={"ID":"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe","Type":"ContainerStarted","Data":"4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa"} Nov 03 02:41:18.243797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:18.243739 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" podStartSLOduration=2.24372422 podStartE2EDuration="2.24372422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:41:18.242066949 +0000 UTC m=+5013.820889539" watchObservedRunningTime="2026-11-03 02:41:18.24372422 +0000 UTC m=+5013.822546806" Nov 03 02:41:19.130413 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:19.130372 4992 generic.go:356] "Generic (PLEG): container finished" podID="6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" containerID="e9d48044f66013992c2159f6dd57883e4c5d874a254f10614a81902840ab7ec9" exitCode=0 Nov 03 02:41:19.130849 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:19.130406 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" event={"ID":"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe","Type":"ContainerDied","Data":"e9d48044f66013992c2159f6dd57883e4c5d874a254f10614a81902840ab7ec9"} Nov 03 02:41:20.289391 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.289365 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:20.358717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.358677 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\") pod \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " Nov 03 02:41:20.358932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.358762 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\") pod \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " Nov 03 02:41:20.358932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.358822 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\" (UniqueName: \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\") pod \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\" (UID: \"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe\") " Nov 03 02:41:20.359154 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.359126 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca" pod "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" (UID: "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:20.359284 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.359267 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.361030 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.361004 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca" pod "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" (UID: "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:20.361135 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.361074 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5" pod "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" (UID: "6cc71c7f-ad8d-4296-9cce-6d304a9e18fe"). InnerVolumeSpecName "kube-api-access-2w7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:20.460717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.460678 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:20.460717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:20.460712 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7l5\" (UniqueName: \"kubernetes.io/projected/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe-kube-api-access-2w7l5\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.070232 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.070201 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:41:21.143033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.142943 4992 generic.go:356] "Generic (PLEG): container finished" podID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerID="7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b" exitCode=0 Nov 03 02:41:21.143033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.142999 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" Nov 03 02:41:21.143268 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.143029 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerDied","Data":"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b"} Nov 03 02:41:21.143268 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.143085 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j" event={"ID":"a8acc2db-ccfa-4d0f-b206-c5d40224deae","Type":"ContainerDied","Data":"2d8f1db6ef1be1e8b840b4f439fdfaf7c71594951ebd94dd2a2fae0da14b54af"} Nov 03 02:41:21.143268 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.143113 4992 scope.go:122] "RemoveContainer" containerID="7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b" Nov 03 02:41:21.144468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.144450 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" Nov 03 02:41:21.144468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.144454 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72" event={"ID":"6cc71c7f-ad8d-4296-9cce-6d304a9e18fe","Type":"ContainerDied","Data":"4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa"} Nov 03 02:41:21.144620 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.144498 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5eb0b6702269fd1e82840365fe269282f8138e829a583203437cbd48fd6faa" Nov 03 02:41:21.152627 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.152599 4992 scope.go:122] "RemoveContainer" containerID="330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a" Nov 03 02:41:21.161501 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.161475 4992 scope.go:122] "RemoveContainer" containerID="7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b" Nov 03 02:41:21.161977 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:21.161953 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b\": container with ID starting with 7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b not found: ID does not exist" containerID="7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b" Nov 03 02:41:21.162063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.161989 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b"} err="failed to get container status \"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b\": rpc error: code = NotFound desc = could not find container \"7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b\": container with ID starting with 7ae8019049e6d8ed26446e781cb86a09abd8457e640c1d128a1ecaf30af9518b not found: ID does not exist" Nov 03 02:41:21.162063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.162009 4992 scope.go:122] "RemoveContainer" containerID="330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a" Nov 03 02:41:21.162272 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:21.162248 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a\": container with ID starting with 330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a not found: ID does not exist" containerID="330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a" Nov 03 02:41:21.162320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.162285 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a"} err="failed to get container status \"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a\": rpc error: code = NotFound desc = could not find container \"330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a\": container with ID starting with 330321b7fcbe2af0064434067fc90e99d9b6e6db08ef9c7f8e2b35cb74b16a5a not found: ID does not exist" Nov 03 02:41:21.167708 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167682 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167810 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167728 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167810 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167780 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167810 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167804 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167828 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167847 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.167990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.167966 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.168151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168021 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.168151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168053 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\") pod \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\" (UID: \"a8acc2db-ccfa-4d0f-b206-c5d40224deae\") " Nov 03 02:41:21.168270 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168146 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:21.168323 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168272 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:21.168401 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168372 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:21.168574 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168531 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:21.168700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168590 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.168700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168606 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.168700 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168615 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.168893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.168709 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:21.170287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.170256 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:21.170552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.170529 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:21.170662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.170535 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "kube-api-access-4d4jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:21.170662 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.170587 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo" pod "a8acc2db-ccfa-4d0f-b206-c5d40224deae" (UID: "a8acc2db-ccfa-4d0f-b206-c5d40224deae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:41:21.247425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.247385 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 02:41:21.269150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269117 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.269150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269145 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4jc\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-kube-api-access-4d4jc\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.269150 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269155 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.269442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269164 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.269442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269175 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8acc2db-ccfa-4d0f-b206-c5d40224deae-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.269442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.269183 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8acc2db-ccfa-4d0f-b206-c5d40224deae-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:21.594362 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.594319 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:41:21.597932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:21.597895 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6a5d73-openshift-default-79b4c488bf-b7w7j"] Nov 03 02:41:22.599256 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:22.599217 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" path="/var/lib/kubelet/pods/a8acc2db-ccfa-4d0f-b206-c5d40224deae/volumes" Nov 03 02:41:24.088465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:24.088428 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 02:41:24.088913 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:24.088696 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" containerID="cri-o://be5c4a08ff1867413e0e67a13b0f3ce37fd7bf8592eef9473bb33c2910793978" gracePeriod=30 Nov 03 02:41:28.510964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:28.510926 4992 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 02:41:28.511390 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:28.510987 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 02:41:29.203846 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.203816 4992 generic.go:356] "Generic (PLEG): container finished" podID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerID="be5c4a08ff1867413e0e67a13b0f3ce37fd7bf8592eef9473bb33c2910793978" exitCode=0 Nov 03 02:41:29.203964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.203890 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerDied","Data":"be5c4a08ff1867413e0e67a13b0f3ce37fd7bf8592eef9473bb33c2910793978"} Nov 03 02:41:29.203964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.203960 4992 scope.go:122] "RemoveContainer" containerID="1194fb66238bc601fc5d722db79fc31bafd587ac5e45983d8c9ae4991f048553" Nov 03 02:41:29.351828 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.351762 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 02:41:29.451448 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451411 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451467 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451505 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451592 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451680 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451637 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451682 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451764 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451780 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:29.451983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451847 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.451983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.451880 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\") pod \"8b27a046-e80f-43d0-83de-5cac014ef0bc\" (UID: \"8b27a046-e80f-43d0-83de-5cac014ef0bc\") " Nov 03 02:41:29.452252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452015 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:29.452252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452106 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:29.452252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452166 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:29.452421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452317 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.452421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452338 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.452421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452351 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.452421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452361 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b27a046-e80f-43d0-83de-5cac014ef0bc-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.452421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.452316 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:29.453929 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.453901 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:29.454093 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.454071 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "kube-api-access-km6hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:29.454305 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.454284 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:41:29.454386 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.454291 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token" pod "8b27a046-e80f-43d0-83de-5cac014ef0bc" (UID: "8b27a046-e80f-43d0-83de-5cac014ef0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:29.553702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.553666 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.553702 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.553699 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.554169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.553718 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.554169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.553728 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b27a046-e80f-43d0-83de-5cac014ef0bc-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:29.554169 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:29.553738 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6hx\" (UniqueName: \"kubernetes.io/projected/8b27a046-e80f-43d0-83de-5cac014ef0bc-kube-api-access-km6hx\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:30.210119 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.210090 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" Nov 03 02:41:30.210320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.210136 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2" event={"ID":"8b27a046-e80f-43d0-83de-5cac014ef0bc","Type":"ContainerDied","Data":"e8aa329b9b985705cf6b133ea1a951b672a6d6e7bd894713fe9e7ecfddd81002"} Nov 03 02:41:30.210320 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.210196 4992 scope.go:122] "RemoveContainer" containerID="be5c4a08ff1867413e0e67a13b0f3ce37fd7bf8592eef9473bb33c2910793978" Nov 03 02:41:30.432272 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.432229 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 02:41:30.483996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.483915 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-fmfs2"] Nov 03 02:41:30.594446 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:30.594405 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" path="/var/lib/kubelet/pods/8b27a046-e80f-43d0-83de-5cac014ef0bc/volumes" Nov 03 02:41:32.153825 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:32.153781 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:41:33.571346 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571309 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj"] Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571759 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571774 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571792 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571797 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571812 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:41:33.571823 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571817 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571833 4992 cpu_manager.go:414] "containerMap: removing container" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571838 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acff4567-6d9a-40f4-9349-ac43f440dba4" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571850 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571854 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571865 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" containerName="revoke-keys" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571869 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" containerName="revoke-keys" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571940 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571949 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571959 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b27a046-e80f-43d0-83de-5cac014ef0bc" containerName="istio-proxy" Nov 03 02:41:33.572023 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.571965 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" containerName="revoke-keys" Nov 03 02:41:33.576629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.576611 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.629212 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.629173 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj"] Nov 03 02:41:33.678677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.678640 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:41:33.678875 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.678709 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:41:33.694539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.694493 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.694751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.694554 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.694751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.694656 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csql6\" (UniqueName: \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.795552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.795510 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.795552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.795586 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.795869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.795611 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csql6\" (UniqueName: \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.796206 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.796182 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.798072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.798049 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.858893 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.858815 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csql6\" (UniqueName: \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\") pod \"maas-api-revoke-sub-1e31103d-6lxzj\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:33.889707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:33.889672 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:34.263820 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:34.263795 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj"] Nov 03 02:41:35.247542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:35.247508 4992 generic.go:356] "Generic (PLEG): container finished" podID="e073cda7-db69-4486-9fde-ed69986146c7" containerID="49a94bd04c2efd8e597a264138e51d2ec686ddd4fa6e3d5851ab8102040d09dd" exitCode=0 Nov 03 02:41:35.247990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:35.247590 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" event={"ID":"e073cda7-db69-4486-9fde-ed69986146c7","Type":"ContainerDied","Data":"49a94bd04c2efd8e597a264138e51d2ec686ddd4fa6e3d5851ab8102040d09dd"} Nov 03 02:41:35.247990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:35.247649 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" event={"ID":"e073cda7-db69-4486-9fde-ed69986146c7","Type":"ContainerStarted","Data":"e3ef9f1289c124d8543195944a52e8ceef0b017bf1483d94647093f478e09611"} Nov 03 02:41:36.268675 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.268640 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" containerID="cri-o://9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f" gracePeriod=45 Nov 03 02:41:36.398996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.398969 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:36.513795 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.513725 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 02:41:36.525002 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.524966 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\") pod \"e073cda7-db69-4486-9fde-ed69986146c7\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " Nov 03 02:41:36.525188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.525164 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\") pod \"e073cda7-db69-4486-9fde-ed69986146c7\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " Nov 03 02:41:36.525282 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.525234 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\" (UniqueName: \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\") pod \"e073cda7-db69-4486-9fde-ed69986146c7\" (UID: \"e073cda7-db69-4486-9fde-ed69986146c7\") " Nov 03 02:41:36.525639 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.525604 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca" pod "e073cda7-db69-4486-9fde-ed69986146c7" (UID: "e073cda7-db69-4486-9fde-ed69986146c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:41:36.527175 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.527148 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca" pod "e073cda7-db69-4486-9fde-ed69986146c7" (UID: "e073cda7-db69-4486-9fde-ed69986146c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:36.527960 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.527900 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6" pod "e073cda7-db69-4486-9fde-ed69986146c7" (UID: "e073cda7-db69-4486-9fde-ed69986146c7"). InnerVolumeSpecName "kube-api-access-csql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:36.626376 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626340 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\" (UniqueName: \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626390 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626458 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626498 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626577 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626610 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\") pod \"1a9b988d-9b20-4b02-9982-56335c023ab9\" (UID: \"1a9b988d-9b20-4b02-9982-56335c023ab9\") " Nov 03 02:41:36.626761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626693 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:36.626932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626835 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:36.626932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.626882 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:36.627033 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627007 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csql6\" (UniqueName: \"kubernetes.io/projected/e073cda7-db69-4486-9fde-ed69986146c7-kube-api-access-csql6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.627070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627043 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.627070 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627059 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.627148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627073 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.627148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627089 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.627148 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.627103 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e073cda7-db69-4486-9fde-ed69986146c7-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.628788 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.628767 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:41:36.628928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.628826 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "kube-api-access-bpjmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:41:36.628928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.628827 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs" pod "1a9b988d-9b20-4b02-9982-56335c023ab9" (UID: "1a9b988d-9b20-4b02-9982-56335c023ab9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:41:36.728675 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.728635 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9b988d-9b20-4b02-9982-56335c023ab9-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.728675 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.728669 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9b988d-9b20-4b02-9982-56335c023ab9-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:36.728675 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:36.728680 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjmm\" (UniqueName: \"kubernetes.io/projected/1a9b988d-9b20-4b02-9982-56335c023ab9-kube-api-access-bpjmm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:41:37.260821 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.260790 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" Nov 03 02:41:37.261028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.260782 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj" event={"ID":"e073cda7-db69-4486-9fde-ed69986146c7","Type":"ContainerDied","Data":"e3ef9f1289c124d8543195944a52e8ceef0b017bf1483d94647093f478e09611"} Nov 03 02:41:37.261028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.260924 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ef9f1289c124d8543195944a52e8ceef0b017bf1483d94647093f478e09611" Nov 03 02:41:37.262493 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.262459 4992 generic.go:356] "Generic (PLEG): container finished" podID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerID="9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f" exitCode=0 Nov 03 02:41:37.262660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.262507 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" Nov 03 02:41:37.262660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.262549 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" event={"ID":"1a9b988d-9b20-4b02-9982-56335c023ab9","Type":"ContainerDied","Data":"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f"} Nov 03 02:41:37.262660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.262614 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j" event={"ID":"1a9b988d-9b20-4b02-9982-56335c023ab9","Type":"ContainerDied","Data":"1e9096721611f69660c116b7075b922b01923013a56dec2f21b8fc9d1b52a2d3"} Nov 03 02:41:37.262660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.262638 4992 scope.go:122] "RemoveContainer" containerID="9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f" Nov 03 02:41:37.272347 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.272324 4992 scope.go:122] "RemoveContainer" containerID="9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f" Nov 03 02:41:37.272688 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:41:37.272667 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f\": container with ID starting with 9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f not found: ID does not exist" containerID="9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f" Nov 03 02:41:37.272729 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.272694 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f"} err="failed to get container status \"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f\": rpc error: code = NotFound desc = could not find container \"9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f\": container with ID starting with 9a4205c166d97ac9f9a6bf95d5cc2acd188d31057d9c3b5a5c4033ac7b35509f not found: ID does not exist" Nov 03 02:41:37.433045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.433001 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 02:41:37.484813 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:37.484780 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-w2-9da431-kserve-65b7f6b9f9-2tk6j"] Nov 03 02:41:38.593971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:41:38.593933 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" path="/var/lib/kubelet/pods/1a9b988d-9b20-4b02-9982-56335c023ab9/volumes" Nov 03 02:42:02.151175 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:42:02.151131 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:42:32.151489 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:42:32.151451 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:42:45.337297 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:42:45.337264 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:42:45.340095 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:42:45.340074 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:43:02.150086 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:43:02.150044 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:43:32.151958 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:43:32.151919 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:44:02.150050 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:44:02.149969 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:44:32.151800 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:44:32.151758 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:45:00.159707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.159670 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:00.160210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160102 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e073cda7-db69-4486-9fde-ed69986146c7" containerName="revoke-keys" Nov 03 02:45:00.160210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160114 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e073cda7-db69-4486-9fde-ed69986146c7" containerName="revoke-keys" Nov 03 02:45:00.160210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160128 4992 cpu_manager.go:414] "containerMap: removing container" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" Nov 03 02:45:00.160210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160136 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" Nov 03 02:45:00.160210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160210 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8acc2db-ccfa-4d0f-b206-c5d40224deae" containerName="istio-proxy" Nov 03 02:45:00.160374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160221 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e073cda7-db69-4486-9fde-ed69986146c7" containerName="revoke-keys" Nov 03 02:45:00.160374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.160229 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9b988d-9b20-4b02-9982-56335c023ab9" containerName="main" Nov 03 02:45:00.163350 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.163330 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.243955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.243915 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.244155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.243983 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg85c\" (UniqueName: \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.256204 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.256168 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\"" Nov 03 02:45:00.271868 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.271832 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:00.344779 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.344746 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.344993 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.344820 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg85c\" (UniqueName: \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.345411 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.345388 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.408201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.408153 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg85c\" (UniqueName: \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\") pod \"maas-api-key-cleanup-29894565-9spml\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.476829 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.476777 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:00.848735 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:45:00.848697 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daf9219_86f2_44f3_914a_e2cdad802791.slice/crio-401c4af05997aac8626b4a7623f4b26c2f9b3d86bbbcb8177db08cb989f94034 WatchSource:0}: Error finding container 401c4af05997aac8626b4a7623f4b26c2f9b3d86bbbcb8177db08cb989f94034: Status 404 returned error can't find the container with id 401c4af05997aac8626b4a7623f4b26c2f9b3d86bbbcb8177db08cb989f94034 Nov 03 02:45:00.851016 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:00.850988 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:01.368383 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:01.368338 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerStarted","Data":"72692a7b5c4bff82d072339d4c563fadcccc3989272cefedaff4e8115f549a59"} Nov 03 02:45:01.368808 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:01.368407 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerStarted","Data":"401c4af05997aac8626b4a7623f4b26c2f9b3d86bbbcb8177db08cb989f94034"} Nov 03 02:45:01.489479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:01.489369 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" podStartSLOduration=1.489356083 podStartE2EDuration="1.489356083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:01.487330881 +0000 UTC m=+5237.066153468" watchObservedRunningTime="2026-11-03 02:45:01.489356083 +0000 UTC m=+5237.068178669" Nov 03 02:45:02.153160 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:45:02.153073 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:45:02.376883 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:02.376845 4992 generic.go:356] "Generic (PLEG): container finished" podID="2daf9219-86f2-44f3-914a-e2cdad802791" containerID="72692a7b5c4bff82d072339d4c563fadcccc3989272cefedaff4e8115f549a59" exitCode=60 Nov 03 02:45:02.377305 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:02.376938 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerDied","Data":"72692a7b5c4bff82d072339d4c563fadcccc3989272cefedaff4e8115f549a59"} Nov 03 02:45:02.377305 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:02.377198 4992 scope.go:122] "RemoveContainer" containerID="72692a7b5c4bff82d072339d4c563fadcccc3989272cefedaff4e8115f549a59" Nov 03 02:45:03.388761 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:03.388719 4992 generic.go:356] "Generic (PLEG): container finished" podID="2daf9219-86f2-44f3-914a-e2cdad802791" containerID="b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463" exitCode=60 Nov 03 02:45:03.389220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:03.388912 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerDied","Data":"b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463"} Nov 03 02:45:03.389220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:03.388966 4992 scope.go:122] "RemoveContainer" containerID="72692a7b5c4bff82d072339d4c563fadcccc3989272cefedaff4e8115f549a59" Nov 03 02:45:03.389427 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:03.389407 4992 scope.go:122] "RemoveContainer" containerID="b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463" Nov 03 02:45:03.389795 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:45:03.389770 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-9spml_odh-ai-gateway-infra(2daf9219-86f2-44f3-914a-e2cdad802791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" Nov 03 02:45:04.395295 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:04.395264 4992 scope.go:122] "RemoveContainer" containerID="b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463" Nov 03 02:45:04.395794 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:45:04.395592 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-9spml_odh-ai-gateway-infra(2daf9219-86f2-44f3-914a-e2cdad802791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" Nov 03 02:45:14.590210 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:14.590174 4992 scope.go:122] "RemoveContainer" containerID="b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463" Nov 03 02:45:15.453955 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:15.453916 4992 generic.go:356] "Generic (PLEG): container finished" podID="2daf9219-86f2-44f3-914a-e2cdad802791" containerID="675e3f085b3a388bec95131fdc1993dd77b38cb20b18ac85955ed039ac00bb65" exitCode=60 Nov 03 02:45:15.454166 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:15.453963 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerDied","Data":"675e3f085b3a388bec95131fdc1993dd77b38cb20b18ac85955ed039ac00bb65"} Nov 03 02:45:15.454166 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:15.454010 4992 scope.go:122] "RemoveContainer" containerID="b857d704de07107680a4ce8c7cc9dea1f5299b1bf4ba91e3c39978ca0c9cd463" Nov 03 02:45:15.454412 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:15.454391 4992 scope.go:122] "RemoveContainer" containerID="675e3f085b3a388bec95131fdc1993dd77b38cb20b18ac85955ed039ac00bb65" Nov 03 02:45:15.454781 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:45:15.454756 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894565-9spml_odh-ai-gateway-infra(2daf9219-86f2-44f3-914a-e2cdad802791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" Nov 03 02:45:16.582278 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.582241 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:16.721533 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.721496 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:16.808935 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.808896 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\" (UniqueName: \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\") pod \"2daf9219-86f2-44f3-914a-e2cdad802791\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " Nov 03 02:45:16.809113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.809065 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\") pod \"2daf9219-86f2-44f3-914a-e2cdad802791\" (UID: \"2daf9219-86f2-44f3-914a-e2cdad802791\") " Nov 03 02:45:16.809412 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.809389 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca" pod "2daf9219-86f2-44f3-914a-e2cdad802791" (UID: "2daf9219-86f2-44f3-914a-e2cdad802791"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:16.811114 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.811090 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c" pod "2daf9219-86f2-44f3-914a-e2cdad802791" (UID: "2daf9219-86f2-44f3-914a-e2cdad802791"). InnerVolumeSpecName "kube-api-access-fg85c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:16.910766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.910650 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2daf9219-86f2-44f3-914a-e2cdad802791-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:16.910766 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:16.910707 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg85c\" (UniqueName: \"kubernetes.io/projected/2daf9219-86f2-44f3-914a-e2cdad802791-kube-api-access-fg85c\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:17.468296 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:17.468259 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" Nov 03 02:45:17.468504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:17.468293 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml" event={"ID":"2daf9219-86f2-44f3-914a-e2cdad802791","Type":"ContainerDied","Data":"401c4af05997aac8626b4a7623f4b26c2f9b3d86bbbcb8177db08cb989f94034"} Nov 03 02:45:17.468504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:17.468363 4992 scope.go:122] "RemoveContainer" containerID="675e3f085b3a388bec95131fdc1993dd77b38cb20b18ac85955ed039ac00bb65" Nov 03 02:45:17.637499 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:17.637462 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:17.641657 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:17.641629 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894565-9spml"] Nov 03 02:45:18.595303 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:18.595259 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" path="/var/lib/kubelet/pods/2daf9219-86f2-44f3-914a-e2cdad802791/volumes" Nov 03 02:45:32.149949 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:45:32.149910 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:45:50.417017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.416974 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:45:50.417552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.417288 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" podUID="6457b78d-7602-4ad4-aae5-634f1579c39b" containerName="maas-api" containerID="cri-o://75a764ba2c6b6b51fccb732c03cb6b1edd829026a9058f674fce80eb03ada0e6" gracePeriod=30 Nov 03 02:45:50.649677 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.649636 4992 generic.go:356] "Generic (PLEG): container finished" podID="6457b78d-7602-4ad4-aae5-634f1579c39b" containerID="75a764ba2c6b6b51fccb732c03cb6b1edd829026a9058f674fce80eb03ada0e6" exitCode=0 Nov 03 02:45:50.649877 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.649696 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" event={"ID":"6457b78d-7602-4ad4-aae5-634f1579c39b","Type":"ContainerDied","Data":"75a764ba2c6b6b51fccb732c03cb6b1edd829026a9058f674fce80eb03ada0e6"} Nov 03 02:45:50.684444 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.684367 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:45:50.851114 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.851074 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\") pod \"6457b78d-7602-4ad4-aae5-634f1579c39b\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " Nov 03 02:45:50.851294 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.851155 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\" (UniqueName: \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\") pod \"6457b78d-7602-4ad4-aae5-634f1579c39b\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " Nov 03 02:45:50.851337 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.851321 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\") pod \"6457b78d-7602-4ad4-aae5-634f1579c39b\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " Nov 03 02:45:50.851373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.851360 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\") pod \"6457b78d-7602-4ad4-aae5-634f1579c39b\" (UID: \"6457b78d-7602-4ad4-aae5-634f1579c39b\") " Nov 03 02:45:50.851753 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.851681 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca" pod "6457b78d-7602-4ad4-aae5-634f1579c39b" (UID: "6457b78d-7602-4ad4-aae5-634f1579c39b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:45:50.853426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.853401 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs" pod "6457b78d-7602-4ad4-aae5-634f1579c39b" (UID: "6457b78d-7602-4ad4-aae5-634f1579c39b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:50.853539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.853422 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls" pod "6457b78d-7602-4ad4-aae5-634f1579c39b" (UID: "6457b78d-7602-4ad4-aae5-634f1579c39b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:45:50.853539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.853471 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc" pod "6457b78d-7602-4ad4-aae5-634f1579c39b" (UID: "6457b78d-7602-4ad4-aae5-634f1579c39b"). InnerVolumeSpecName "kube-api-access-2skvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:45:50.952757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.952663 4992 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-metrics-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:50.952757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.952695 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skvc\" (UniqueName: \"kubernetes.io/projected/6457b78d-7602-4ad4-aae5-634f1579c39b-kube-api-access-2skvc\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:50.952757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.952706 4992 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6457b78d-7602-4ad4-aae5-634f1579c39b-openshift-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:50.952757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:50.952718 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6457b78d-7602-4ad4-aae5-634f1579c39b-maas-api-tls\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:45:51.655845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:51.655813 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" Nov 03 02:45:51.656286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:51.655821 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s" event={"ID":"6457b78d-7602-4ad4-aae5-634f1579c39b","Type":"ContainerDied","Data":"a496003374ed7d3d4871afd2036a50fd75a9a3622dcb64de58b98c9040ef9c5e"} Nov 03 02:45:51.656286 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:51.655944 4992 scope.go:122] "RemoveContainer" containerID="75a764ba2c6b6b51fccb732c03cb6b1edd829026a9058f674fce80eb03ada0e6" Nov 03 02:45:51.828975 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:51.828939 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:45:51.880977 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:51.880940 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f0391a-86d5577ccf-cx64s"] Nov 03 02:45:52.595109 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:45:52.595067 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6457b78d-7602-4ad4-aae5-634f1579c39b" path="/var/lib/kubelet/pods/6457b78d-7602-4ad4-aae5-634f1579c39b/volumes" Nov 03 02:46:02.150544 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:02.150507 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:46:03.123345 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:03.123302 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 02:46:03.174996 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:03.174959 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 02:46:03.176825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:03.176792 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 02:46:03.179861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:03.179835 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 02:46:18.143456 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.143411 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" containerID="cri-o://8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a" gracePeriod=45 Nov 03 02:46:18.195048 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.195006 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" containerID="cri-o://067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583" gracePeriod=45 Nov 03 02:46:18.195801 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.195756 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" containerID="cri-o://c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2" gracePeriod=45 Nov 03 02:46:18.198762 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.198728 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" containerID="cri-o://9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96" gracePeriod=45 Nov 03 02:46:18.351585 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.351486 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused" Nov 03 02:46:18.462820 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.462758 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 02:46:18.510759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.510733 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 02:46:18.517178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.517149 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 02:46:18.522209 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.522179 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.522409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.522391 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\" (UniqueName: \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.522665 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.522648 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.522861 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.522846 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.523031 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.523019 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.523133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.523120 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\") pod \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\" (UID: \"a7409933-d0c1-4bb6-b00c-e30749baf8ae\") " Nov 03 02:46:18.526723 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.526347 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.526723 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.526472 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:18.526917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.526733 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.526917 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.526911 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.527240 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.527214 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 02:46:18.529258 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.529231 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "kube-api-access-lw7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:18.533156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.533123 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm" pod "a7409933-d0c1-4bb6-b00c-e30749baf8ae" (UID: "a7409933-d0c1-4bb6-b00c-e30749baf8ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.624696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624653 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.624902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624724 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.624902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624765 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.624902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624800 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.624902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624858 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\" (UniqueName: \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.624902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624889 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624921 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624950 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.624996 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625052 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625045 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625080 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625130 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625162 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625203 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.625193 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625233 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625275 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\" (UniqueName: \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625320 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\" (UniqueName: \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\") pod \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\" (UID: \"85e1494f-7f5e-4886-88c4-1e68effbfbb1\") " Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625353 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625359 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\") pod \"5729d0ae-797e-40ad-85d3-9600a13ade3f\" (UID: \"5729d0ae-797e-40ad-85d3-9600a13ade3f\") " Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625373 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.625831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625587 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.626165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625835 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\") pod \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\" (UID: \"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b\") " Nov 03 02:46:18.626165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.625867 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.626165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626091 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.626165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626119 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.626389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626321 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626342 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626358 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626372 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7409933-d0c1-4bb6-b00c-e30749baf8ae-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626389 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626385 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626398 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626411 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626424 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626437 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw7lt\" (UniqueName: \"kubernetes.io/projected/a7409933-d0c1-4bb6-b00c-e30749baf8ae-kube-api-access-lw7lt\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626454 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626468 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626337 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626482 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626518 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.626715 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.626536 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7409933-d0c1-4bb6-b00c-e30749baf8ae-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.628259 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.628230 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.628468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.628437 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:18.628647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.628517 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "kube-api-access-8lf9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:18.629004 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.628978 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.629224 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.629200 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "kube-api-access-qjpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:18.629425 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.629403 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:18.629594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.629569 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd" pod "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" (UID: "bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b"). InnerVolumeSpecName "kube-api-access-kkjcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:18.629897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.629877 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm" pod "5729d0ae-797e-40ad-85d3-9600a13ade3f" (UID: "5729d0ae-797e-40ad-85d3-9600a13ade3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:18.630225 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.630212 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs" pod "85e1494f-7f5e-4886-88c4-1e68effbfbb1" (UID: "85e1494f-7f5e-4886-88c4-1e68effbfbb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:46:18.727922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727829 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lf9j\" (UniqueName: \"kubernetes.io/projected/5729d0ae-797e-40ad-85d3-9600a13ade3f-kube-api-access-8lf9j\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.727922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727877 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85e1494f-7f5e-4886-88c4-1e68effbfbb1-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.727922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727892 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.727922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727905 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5729d0ae-797e-40ad-85d3-9600a13ade3f-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.727922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727918 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.728202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727931 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjcd\" (UniqueName: \"kubernetes.io/projected/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-kube-api-access-kkjcd\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.728202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727945 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjpqz\" (UniqueName: \"kubernetes.io/projected/85e1494f-7f5e-4886-88c4-1e68effbfbb1-kube-api-access-qjpqz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.728202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727958 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.728202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727972 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5729d0ae-797e-40ad-85d3-9600a13ade3f-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.728202 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.727984 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85e1494f-7f5e-4886-88c4-1e68effbfbb1-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:18.805324 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.805277 4992 generic.go:356] "Generic (PLEG): container finished" podID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerID="067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583" exitCode=0 Nov 03 02:46:18.805542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.805347 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" Nov 03 02:46:18.805542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.805357 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" event={"ID":"a7409933-d0c1-4bb6-b00c-e30749baf8ae","Type":"ContainerDied","Data":"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583"} Nov 03 02:46:18.805542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.805406 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9" event={"ID":"a7409933-d0c1-4bb6-b00c-e30749baf8ae","Type":"ContainerDied","Data":"f79d95d22d33fc15a7d95767132c1edcd7f2405108a2fa8c17f6a176dee85362"} Nov 03 02:46:18.805542 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.805429 4992 scope.go:122] "RemoveContainer" containerID="067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583" Nov 03 02:46:18.807323 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.807291 4992 generic.go:356] "Generic (PLEG): container finished" podID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerID="c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2" exitCode=0 Nov 03 02:46:18.807442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.807343 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" Nov 03 02:46:18.807442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.807407 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" event={"ID":"5729d0ae-797e-40ad-85d3-9600a13ade3f","Type":"ContainerDied","Data":"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2"} Nov 03 02:46:18.807629 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.807449 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz" event={"ID":"5729d0ae-797e-40ad-85d3-9600a13ade3f","Type":"ContainerDied","Data":"92f62124b9afd2df09eb118176529ec9cb9a70d52f6a4830a43f1d36bd026a02"} Nov 03 02:46:18.809018 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.808998 4992 generic.go:356] "Generic (PLEG): container finished" podID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerID="9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96" exitCode=0 Nov 03 02:46:18.809133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.809045 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" Nov 03 02:46:18.809133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.809081 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" event={"ID":"85e1494f-7f5e-4886-88c4-1e68effbfbb1","Type":"ContainerDied","Data":"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96"} Nov 03 02:46:18.809133 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.809112 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp" event={"ID":"85e1494f-7f5e-4886-88c4-1e68effbfbb1","Type":"ContainerDied","Data":"0b9e2eac8d5f851daa98c3ff4fa9552baad89fedf24e444f452cd0886837f38f"} Nov 03 02:46:18.810751 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.810726 4992 generic.go:356] "Generic (PLEG): container finished" podID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerID="8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a" exitCode=0 Nov 03 02:46:18.810880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.810778 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" Nov 03 02:46:18.810880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.810822 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" event={"ID":"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b","Type":"ContainerDied","Data":"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a"} Nov 03 02:46:18.810880 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.810859 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z" event={"ID":"bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b","Type":"ContainerDied","Data":"38220b22adf8a5e11166939b8f64e817aa3859f248332e9f569e0be207206e47"} Nov 03 02:46:18.816971 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.816951 4992 scope.go:122] "RemoveContainer" containerID="067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583" Nov 03 02:46:18.817298 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:18.817273 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583\": container with ID starting with 067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583 not found: ID does not exist" containerID="067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583" Nov 03 02:46:18.817379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.817310 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583"} err="failed to get container status \"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583\": rpc error: code = NotFound desc = could not find container \"067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583\": container with ID starting with 067ca2a0b1a8745802637b3c76d23011c40a47ceed88011e171a7189964f6583 not found: ID does not exist" Nov 03 02:46:18.817379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.817334 4992 scope.go:122] "RemoveContainer" containerID="c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2" Nov 03 02:46:18.829367 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.829339 4992 scope.go:122] "RemoveContainer" containerID="c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2" Nov 03 02:46:18.829793 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:18.829762 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2\": container with ID starting with c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2 not found: ID does not exist" containerID="c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2" Nov 03 02:46:18.829907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.829803 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2"} err="failed to get container status \"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2\": rpc error: code = NotFound desc = could not find container \"c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2\": container with ID starting with c68bbdd132a70a188163503ddf78df8efd8771f2c7d7b93740672f63f582d9c2 not found: ID does not exist" Nov 03 02:46:18.829907 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.829831 4992 scope.go:122] "RemoveContainer" containerID="9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96" Nov 03 02:46:18.841936 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.841905 4992 scope.go:122] "RemoveContainer" containerID="9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96" Nov 03 02:46:18.842336 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:18.842308 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96\": container with ID starting with 9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96 not found: ID does not exist" containerID="9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96" Nov 03 02:46:18.842432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.842349 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96"} err="failed to get container status \"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96\": rpc error: code = NotFound desc = could not find container \"9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96\": container with ID starting with 9b1c77f6f45b9a14bc902cffc7fa952265d4e0ebfb797d0c043e323373b57a96 not found: ID does not exist" Nov 03 02:46:18.842432 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.842378 4992 scope.go:122] "RemoveContainer" containerID="8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a" Nov 03 02:46:18.853154 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.853129 4992 scope.go:122] "RemoveContainer" containerID="8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a" Nov 03 02:46:18.853522 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:18.853497 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a\": container with ID starting with 8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a not found: ID does not exist" containerID="8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a" Nov 03 02:46:18.853632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:18.853533 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a"} err="failed to get container status \"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a\": rpc error: code = NotFound desc = could not find container \"8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a\": container with ID starting with 8079298f27d2f8c94ac4f865fd8542e048c1e2ec985700d2f2ba5d69afd2b09a not found: ID does not exist" Nov 03 02:46:19.061647 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.061542 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 02:46:19.071903 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.071868 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/simulator-w0-b6fb9b-kserve-5b5d8867bb-fztmp"] Nov 03 02:46:19.184192 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.184157 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 02:46:19.188017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.187989 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-2-w0-b6fb9b-kserve-948b58dc7-ngg6z"] Nov 03 02:46:19.301024 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.300990 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 02:46:19.304593 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.304568 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/distinct-w0-b6fb9b-kserve-75cf685c8b-wbvm9"] Nov 03 02:46:19.417418 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.417380 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 02:46:19.421370 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:19.421336 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b6fb9b/premium-w0-b6fb9b-kserve-685b77794c-tlhcz"] Nov 03 02:46:20.594181 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:20.594146 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" path="/var/lib/kubelet/pods/5729d0ae-797e-40ad-85d3-9600a13ade3f/volumes" Nov 03 02:46:20.594595 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:20.594577 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" path="/var/lib/kubelet/pods/85e1494f-7f5e-4886-88c4-1e68effbfbb1/volumes" Nov 03 02:46:20.594988 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:20.594972 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" path="/var/lib/kubelet/pods/a7409933-d0c1-4bb6-b00c-e30749baf8ae/volumes" Nov 03 02:46:20.595631 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:20.595618 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" path="/var/lib/kubelet/pods/bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b/volumes" Nov 03 02:46:21.049419 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:21.049380 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72"] Nov 03 02:46:21.101074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:21.101034 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8273e395-whv72"] Nov 03 02:46:22.593590 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:22.593524 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc71c7f-ad8d-4296-9cce-6d304a9e18fe" path="/var/lib/kubelet/pods/6cc71c7f-ad8d-4296-9cce-6d304a9e18fe/volumes" Nov 03 02:46:32.149876 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:32.149833 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:46:34.329579 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.329524 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:34.330042 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330023 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330043 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330052 4992 cpu_manager.go:414] "containerMap: removing container" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330057 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330067 4992 cpu_manager.go:414] "containerMap: removing container" podUID="6457b78d-7602-4ad4-aae5-634f1579c39b" containerName="maas-api" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330073 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6457b78d-7602-4ad4-aae5-634f1579c39b" containerName="maas-api" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330086 4992 cpu_manager.go:414] "containerMap: removing container" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" Nov 03 02:46:34.330091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330092 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330100 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330109 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330119 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330124 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330132 4992 cpu_manager.go:414] "containerMap: removing container" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330136 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330154 4992 cpu_manager.go:414] "containerMap: removing container" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330159 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330222 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7409933-d0c1-4bb6-b00c-e30749baf8ae" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330234 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330239 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="5729d0ae-797e-40ad-85d3-9600a13ade3f" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330245 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="6457b78d-7602-4ad4-aae5-634f1579c39b" containerName="maas-api" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330251 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e1494f-7f5e-4886-88c4-1e68effbfbb1" containerName="main" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330259 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:46:34.330318 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.330267 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdb8b33-8fe5-4814-9a7e-02a8f0247d4b" containerName="main" Nov 03 02:46:34.333684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.333664 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.388098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.388066 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\"" Nov 03 02:46:34.441649 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.441607 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:34.480777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.480730 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jjr\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.480777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.480783 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.480909 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.480972 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481060 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.481049 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.481098 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.481120 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.481153 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.481194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.481172 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582063 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582118 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582137 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582159 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jjr\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582183 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582221 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582253 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582422 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582486 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582464 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582853 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582546 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.582941 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582917 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.583020 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.582961 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.583073 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.583018 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.583537 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.583512 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.584619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.584599 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.584919 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.584900 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.645983 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.645945 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.646126 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.646071 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jjr\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:34.946822 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:34.946778 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:35.373468 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:46:35.373414 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc705cf2_cb5d_468b_9e80_a25000d84db6.slice/crio-8c4652a3494a15b5d9c1b59a78eb4164368305d625c884d992ea0e1ce5425b51 WatchSource:0}: Error finding container 8c4652a3494a15b5d9c1b59a78eb4164368305d625c884d992ea0e1ce5425b51: Status 404 returned error can't find the container with id 8c4652a3494a15b5d9c1b59a78eb4164368305d625c884d992ea0e1ce5425b51 Nov 03 02:46:35.375619 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.375589 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:35.375911 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.375876 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:35.375993 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.375959 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:35.376053 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.376010 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:46:35.901858 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.901815 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" event={"ID":"fc705cf2-cb5d-468b-9e80-a25000d84db6","Type":"ContainerStarted","Data":"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7"} Nov 03 02:46:35.902055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.901885 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" event={"ID":"fc705cf2-cb5d-468b-9e80-a25000d84db6","Type":"ContainerStarted","Data":"8c4652a3494a15b5d9c1b59a78eb4164368305d625c884d992ea0e1ce5425b51"} Nov 03 02:46:35.946915 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:35.946855 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:36.948939 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:36.948904 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:46:36.949374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:36.948961 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 02:46:37.048966 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:37.048909 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podStartSLOduration=3.048888631 podStartE2EDuration="3.048888631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:36.086861649 +0000 UTC m=+5331.665684237" watchObservedRunningTime="2026-11-03 02:46:37.048888631 +0000 UTC m=+5332.627711218" Nov 03 02:46:37.050186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:37.050166 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj"] Nov 03 02:46:37.058717 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:37.058683 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e31103d-6lxzj"] Nov 03 02:46:37.947912 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:37.947866 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:46:37.948111 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:37.947928 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 02:46:38.594381 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:38.594340 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e073cda7-db69-4486-9fde-ed69986146c7" path="/var/lib/kubelet/pods/e073cda7-db69-4486-9fde-ed69986146c7/volumes" Nov 03 02:46:38.947407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:38.947367 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:46:38.947621 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:38.947437 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 02:46:39.947220 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:39.947188 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:46:39.947721 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:39.947695 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 02:46:40.947782 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:40.947743 4992 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 02:46:40.948280 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:40.947805 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 02:46:41.952494 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:41.952458 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:41.952988 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:41.952891 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:41.953912 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:41.953887 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:43.943297 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:43.943258 4992 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" secret="" err="secret \"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\" not found" Nov 03 02:46:43.979922 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:43.979878 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:44.947630 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:44.947578 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" containerID="cri-o://65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7" gracePeriod=30 Nov 03 02:46:45.772826 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:45.772789 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nhqsx_22284cad-4990-4345-ae90-b13bdc75e40a/manager/0.log" Nov 03 02:46:46.327177 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:46.327144 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwl4r_9ffc6632-9a8c-46a2-8493-77016e28e030/kuadrant-console-plugin/0.log" Nov 03 02:46:48.132017 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:48.131979 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8f9fb_8b089aeb-b68c-4a95-b72e-daa34a2d0886/manager/0.log" Nov 03 02:46:49.683869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:49.683835 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/2.log" Nov 03 02:46:50.205776 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.205729 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:50.224779 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224690 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.224779 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224744 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224783 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224808 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224853 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224877 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225037 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.224917 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225053 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225125 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\") pod \"fc705cf2-cb5d-468b-9e80-a25000d84db6\" (UID: \"fc705cf2-cb5d-468b-9e80-a25000d84db6\") " Nov 03 02:46:50.225321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225184 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:50.225321 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225210 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:50.225534 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225404 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:50.225534 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225506 4992 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-credential-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.225534 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225518 4992 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.225534 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225528 4992 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-data\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.225827 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225649 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:46:50.225883 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.225832 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:50.227498 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.227463 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:46:50.227498 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.227479 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 02:46:50.227727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.227517 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "kube-api-access-77jjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:50.227727 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.227536 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token" pod "fc705cf2-cb5d-468b-9e80-a25000d84db6" (UID: "fc705cf2-cb5d-468b-9e80-a25000d84db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:46:50.282200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.282165 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sjhd2_289f89f9-3d93-430e-9df8-0600cdd9b040/manager/0.log" Nov 03 02:46:50.326186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326148 4992 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-podinfo\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.326186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326185 4992 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-token\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.326423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326197 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jjr\" (UniqueName: \"kubernetes.io/projected/fc705cf2-cb5d-468b-9e80-a25000d84db6-kube-api-access-77jjr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.326423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326208 4992 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc705cf2-cb5d-468b-9e80-a25000d84db6-istiod-ca-cert\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.326423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326217 4992 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-istio-envoy\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.326423 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.326226 4992 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc705cf2-cb5d-468b-9e80-a25000d84db6-workload-socket\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:46:50.979113 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.979076 4992 generic.go:356] "Generic (PLEG): container finished" podID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerID="65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7" exitCode=0 Nov 03 02:46:50.979530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.979116 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" Nov 03 02:46:50.979530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.979162 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" event={"ID":"fc705cf2-cb5d-468b-9e80-a25000d84db6","Type":"ContainerDied","Data":"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7"} Nov 03 02:46:50.979530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.979220 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw" event={"ID":"fc705cf2-cb5d-468b-9e80-a25000d84db6","Type":"ContainerDied","Data":"8c4652a3494a15b5d9c1b59a78eb4164368305d625c884d992ea0e1ce5425b51"} Nov 03 02:46:50.979530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.979234 4992 scope.go:122] "RemoveContainer" containerID="65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7" Nov 03 02:46:50.988287 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.988268 4992 scope.go:122] "RemoveContainer" containerID="65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7" Nov 03 02:46:50.988576 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:46:50.988545 4992 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7\": container with ID starting with 65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7 not found: ID does not exist" containerID="65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7" Nov 03 02:46:50.988624 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:50.988584 4992 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7"} err="failed to get container status \"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7\": rpc error: code = NotFound desc = could not find container \"65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7\": container with ID starting with 65816c91456649ad8b5b3ddd44e322a32b958a7e46d866c9b29a7b80af14cbe7 not found: ID does not exist" Nov 03 02:46:51.150039 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:51.150001 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:51.202185 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:51.202148 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-9stdw"] Nov 03 02:46:51.520984 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:51.520951 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6nfb8_8b35e9d1-25de-4f51-af84-1b3f6a965f6c/server/0.log" Nov 03 02:46:51.768633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:51.768599 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vpzrm_8972b343-b983-4b3c-879e-d4a6105ad81c/manager/0.log" Nov 03 02:46:52.594450 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:52.594414 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" path="/var/lib/kubelet/pods/fc705cf2-cb5d-468b-9e80-a25000d84db6/volumes" Nov 03 02:46:52.831821 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:46:52.831772 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/1.log" Nov 03 02:47:02.151002 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:47:02.150963 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:47:32.151540 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:47:32.151499 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:47:38.632684 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.632649 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6"] Nov 03 02:47:38.633097 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.633085 4992 cpu_manager.go:414] "containerMap: removing container" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" Nov 03 02:47:38.633141 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.633100 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" Nov 03 02:47:38.633187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.633177 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2daf9219-86f2-44f3-914a-e2cdad802791" containerName="cleanup" Nov 03 02:47:38.633230 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.633190 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc705cf2-cb5d-468b-9e80-a25000d84db6" containerName="istio-proxy" Nov 03 02:47:38.636417 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.636394 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.678882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.678842 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.679091 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.679075 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.679160 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.679148 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94kr\" (UniqueName: \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.691364 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.691329 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6"] Nov 03 02:47:38.723370 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.723332 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:47:38.723370 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.723343 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:47:38.723671 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.723370 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:47:38.740915 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.740886 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 02:47:38.780180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.780145 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94kr\" (UniqueName: \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.780368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.780203 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.780368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.780248 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.781049 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.781021 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.782841 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.782818 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.843300 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.843263 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94kr\" (UniqueName: \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\") pod \"maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:38.949157 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:38.949126 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:39.321256 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:47:39.321220 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a401b4_c3c9_4be8_90ea_e862a819c70e.slice/crio-d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40 WatchSource:0}: Error finding container d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40: Status 404 returned error can't find the container with id d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40 Nov 03 02:47:39.322793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:39.322771 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6"] Nov 03 02:47:40.237420 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:40.237389 4992 generic.go:356] "Generic (PLEG): container finished" podID="14a401b4-c3c9-4be8-90ea-e862a819c70e" containerID="0b7d8f1abebce826d05762a8bd0e46fe762d6211249d2b1151beac02fedeac21" exitCode=0 Nov 03 02:47:40.237842 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:40.237473 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" event={"ID":"14a401b4-c3c9-4be8-90ea-e862a819c70e","Type":"ContainerDied","Data":"0b7d8f1abebce826d05762a8bd0e46fe762d6211249d2b1151beac02fedeac21"} Nov 03 02:47:40.237842 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:40.237529 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" event={"ID":"14a401b4-c3c9-4be8-90ea-e862a819c70e","Type":"ContainerStarted","Data":"d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40"} Nov 03 02:47:41.387666 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.387635 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:41.507447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.507363 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\" (UniqueName: \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\") pod \"14a401b4-c3c9-4be8-90ea-e862a819c70e\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " Nov 03 02:47:41.507718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.507492 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\") pod \"14a401b4-c3c9-4be8-90ea-e862a819c70e\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " Nov 03 02:47:41.507718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.507535 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\") pod \"14a401b4-c3c9-4be8-90ea-e862a819c70e\" (UID: \"14a401b4-c3c9-4be8-90ea-e862a819c70e\") " Nov 03 02:47:41.508188 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.508157 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca" pod "14a401b4-c3c9-4be8-90ea-e862a819c70e" (UID: "14a401b4-c3c9-4be8-90ea-e862a819c70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:47:41.510215 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.510191 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr" pod "14a401b4-c3c9-4be8-90ea-e862a819c70e" (UID: "14a401b4-c3c9-4be8-90ea-e862a819c70e"). InnerVolumeSpecName "kube-api-access-r94kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:47:41.510373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.510356 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca" pod "14a401b4-c3c9-4be8-90ea-e862a819c70e" (UID: "14a401b4-c3c9-4be8-90ea-e862a819c70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:47:41.608845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.608809 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:47:41.608845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.608840 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14a401b4-c3c9-4be8-90ea-e862a819c70e-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:47:41.608845 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:41.608853 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94kr\" (UniqueName: \"kubernetes.io/projected/14a401b4-c3c9-4be8-90ea-e862a819c70e-kube-api-access-r94kr\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:47:42.249082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:42.249044 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" event={"ID":"14a401b4-c3c9-4be8-90ea-e862a819c70e","Type":"ContainerDied","Data":"d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40"} Nov 03 02:47:42.249082 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:42.249074 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6" Nov 03 02:47:42.249356 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:42.249096 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9219b910f5b34168110d86c61ca83d073931c1ee76ee003f8eaf6f4f7fcdf40" Nov 03 02:47:45.369409 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:45.369289 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:47:45.380273 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:45.375962 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:47:46.174836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:46.174798 4992 scope.go:122] "RemoveContainer" containerID="49a94bd04c2efd8e597a264138e51d2ec686ddd4fa6e3d5851ab8102040d09dd" Nov 03 02:47:46.183549 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:46.183526 4992 scope.go:122] "RemoveContainer" containerID="e9d48044f66013992c2159f6dd57883e4c5d874a254f10614a81902840ab7ec9" Nov 03 02:47:58.580122 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:58.580085 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6"] Nov 03 02:47:58.631637 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:47:58.631595 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82d1bf6f-x4ft6"] Nov 03 02:48:00.593743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:00.593699 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a401b4-c3c9-4be8-90ea-e862a819c70e" path="/var/lib/kubelet/pods/14a401b4-c3c9-4be8-90ea-e862a819c70e/volumes" Nov 03 02:48:02.150642 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:48:02.150608 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:48:31.238047 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.238008 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp"] Nov 03 02:48:31.238614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.238411 4992 cpu_manager.go:414] "containerMap: removing container" podUID="14a401b4-c3c9-4be8-90ea-e862a819c70e" containerName="revoke-keys" Nov 03 02:48:31.238614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.238422 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a401b4-c3c9-4be8-90ea-e862a819c70e" containerName="revoke-keys" Nov 03 02:48:31.238614 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.238497 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a401b4-c3c9-4be8-90ea-e862a819c70e" containerName="revoke-keys" Nov 03 02:48:31.241718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.241689 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.296600 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.296541 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\"" Nov 03 02:48:31.363551 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.363513 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp"] Nov 03 02:48:31.365176 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365144 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365449 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365417 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0277aa76-953c-4912-861b-c1e841b0a47e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365598 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365496 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4btz\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-kube-api-access-c4btz\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365598 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365544 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365778 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365719 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365778 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365759 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0277aa76-953c-4912-861b-c1e841b0a47e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365792 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365818 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.365902 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.365846 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467141 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0277aa76-953c-4912-861b-c1e841b0a47e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467190 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4btz\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-kube-api-access-c4btz\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467225 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467266 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467285 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0277aa76-953c-4912-861b-c1e841b0a47e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467379 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467318 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467442 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467468 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467507 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467869 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467839 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.467973 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467948 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.468092 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.467948 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.468092 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.468016 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.468257 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.468237 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0277aa76-953c-4912-861b-c1e841b0a47e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.469777 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.469758 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0277aa76-953c-4912-861b-c1e841b0a47e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.469904 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.469884 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0277aa76-953c-4912-861b-c1e841b0a47e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.530963 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.530887 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.531186 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.531156 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4btz\" (UniqueName: \"kubernetes.io/projected/0277aa76-953c-4912-861b-c1e841b0a47e-kube-api-access-c4btz\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nf7vp\" (UID: \"0277aa76-953c-4912-861b-c1e841b0a47e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.555084 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.555049 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:31.977624 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:48:31.977587 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0277aa76_953c_4912_861b_c1e841b0a47e.slice/crio-d47df9a78d606780a9c701d5656f49538ca3a3d2a3d401e6e9cb956af8ccba8e WatchSource:0}: Error finding container d47df9a78d606780a9c701d5656f49538ca3a3d2a3d401e6e9cb956af8ccba8e: Status 404 returned error can't find the container with id d47df9a78d606780a9c701d5656f49538ca3a3d2a3d401e6e9cb956af8ccba8e Nov 03 02:48:31.979407 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.979283 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp"] Nov 03 02:48:31.980101 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.980063 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:31.980187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.980128 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:31.980187 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:31.980157 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:48:32.150571 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:48:32.150516 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:48:32.503398 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:32.503353 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" event={"ID":"0277aa76-953c-4912-861b-c1e841b0a47e","Type":"ContainerStarted","Data":"df47bccd4a0020ecb828f2b8d64725910054204e7a77d1f9116a24555e3e8078"} Nov 03 02:48:32.503859 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:32.503452 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" event={"ID":"0277aa76-953c-4912-861b-c1e841b0a47e","Type":"ContainerStarted","Data":"d47df9a78d606780a9c701d5656f49538ca3a3d2a3d401e6e9cb956af8ccba8e"} Nov 03 02:48:32.555659 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:32.555623 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:32.628752 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:32.628696 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podStartSLOduration=1.6286821009999999 podStartE2EDuration="1.628682101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:32.628079274 +0000 UTC m=+5448.206901861" watchObservedRunningTime="2026-11-03 02:48:32.628682101 +0000 UTC m=+5448.207504687" Nov 03 02:48:33.557455 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:33.557417 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:33.557889 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:33.557477 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:34.556290 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:34.556250 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:34.556513 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:34.556319 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:35.556072 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:35.556033 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:35.556479 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:35.556096 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:36.556074 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:36.556035 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:36.556587 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:36.556095 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:37.556434 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:37.556390 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:37.556874 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:37.556451 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:38.555769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:38.555723 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:38.556009 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:38.555794 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:39.556090 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:39.556051 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 02:48:39.556519 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:39.556113 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 02:48:41.596851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:41.596794 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:48:41.597719 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:41.596873 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:48:42.495932 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:42.495893 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:42.496223 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:42.496188 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:48:42.497098 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:48:42.497080 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" Nov 03 02:49:02.150617 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:49:02.150579 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:49:31.688516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.688479 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf"] Nov 03 02:49:31.692028 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.692010 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863194 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863242 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863342 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863395 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863414 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33773d82-e52c-4603-899c-6935a2db93df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863439 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33773d82-e52c-4603-899c-6935a2db93df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863494 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863534 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.863611 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.863569 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nskf\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-kube-api-access-9nskf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.892156 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.892119 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf"] Nov 03 02:49:31.965178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965093 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965136 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965178 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965158 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33773d82-e52c-4603-899c-6935a2db93df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965271 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33773d82-e52c-4603-899c-6935a2db93df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965330 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965369 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965465 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965402 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nskf\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-kube-api-access-9nskf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965494 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965532 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965552 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.965743 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965724 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.966003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965896 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.966003 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.965899 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.966121 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.966074 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33773d82-e52c-4603-899c-6935a2db93df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.968029 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.968003 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33773d82-e52c-4603-899c-6935a2db93df-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:31.968329 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:31.968307 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33773d82-e52c-4603-899c-6935a2db93df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:32.028764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.028734 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nskf\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-kube-api-access-9nskf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:32.028928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.028733 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33773d82-e52c-4603-899c-6935a2db93df-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nd2gf\" (UID: \"33773d82-e52c-4603-899c-6935a2db93df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:32.150259 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:49:32.150220 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:49:32.305063 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.304973 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:32.730334 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:49:32.730302 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33773d82_e52c_4603_899c_6935a2db93df.slice/crio-27246cc04f725bb298bd989a3e33f91c9f40d46b6f6801dbe37105150770d5b7 WatchSource:0}: Error finding container 27246cc04f725bb298bd989a3e33f91c9f40d46b6f6801dbe37105150770d5b7: Status 404 returned error can't find the container with id 27246cc04f725bb298bd989a3e33f91c9f40d46b6f6801dbe37105150770d5b7 Nov 03 02:49:32.732592 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.732540 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf"] Nov 03 02:49:32.732791 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.732759 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:32.732840 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.732827 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:32.732876 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.732857 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:32.804787 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.804751 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" event={"ID":"33773d82-e52c-4603-899c-6935a2db93df","Type":"ContainerStarted","Data":"f88b758af662d7ff749a2ce0d218c019b54a2b3cfafc031e3958adbc3939f750"} Nov 03 02:49:32.804964 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.804809 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" event={"ID":"33773d82-e52c-4603-899c-6935a2db93df","Type":"ContainerStarted","Data":"27246cc04f725bb298bd989a3e33f91c9f40d46b6f6801dbe37105150770d5b7"} Nov 03 02:49:32.931904 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:32.931847 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podStartSLOduration=1.931831114 podStartE2EDuration="1.931831114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:32.930279374 +0000 UTC m=+5508.509101961" watchObservedRunningTime="2026-11-03 02:49:32.931831114 +0000 UTC m=+5508.510653699" Nov 03 02:49:33.305796 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:33.305735 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:33.307594 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:33.307543 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:33.307769 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:33.307610 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:34.306222 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:34.306176 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:34.306745 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:34.306247 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:35.306278 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:35.306232 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:35.306734 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:35.306308 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:36.305796 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:36.305735 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:36.305991 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:36.305817 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:37.306251 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:37.306205 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:37.306699 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:37.306277 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:38.306147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:38.306104 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:38.306332 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:38.306184 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:39.305841 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:39.305784 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 03 02:49:39.306071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:39.305859 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 03 02:49:41.305957 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:41.305915 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:41.306508 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:41.305984 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:42.347775 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:42.347734 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:49:42.348189 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:42.347808 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:49:43.101525 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:43.101485 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:43.102340 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:43.102306 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:43.102871 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:43.102847 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" Nov 03 02:49:46.840252 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.840217 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn"] Nov 03 02:49:46.844201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.844182 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.904006 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.903972 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn"] Nov 03 02:49:46.912200 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912172 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912343 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912217 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912343 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912312 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912454 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912427 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912494 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912531 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdbd\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-kube-api-access-8bdbd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912554 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912567 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb331db-b6cd-4bce-a099-f918904eec77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912623 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb331db-b6cd-4bce-a099-f918904eec77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:46.912653 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:46.912643 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013198 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013157 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013209 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013320 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdbd\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-kube-api-access-8bdbd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013354 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb331db-b6cd-4bce-a099-f918904eec77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013396 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb331db-b6cd-4bce-a099-f918904eec77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013421 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013456 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013508 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013581 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013544 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013947 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013598 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.013947 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013887 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.014046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.013955 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.014046 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.014003 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.014285 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.014251 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb331db-b6cd-4bce-a099-f918904eec77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.016126 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.016081 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb331db-b6cd-4bce-a099-f918904eec77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.016242 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.016222 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb331db-b6cd-4bce-a099-f918904eec77-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.082524 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.082483 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.082765 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.082746 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdbd\" (UniqueName: \"kubernetes.io/projected/5fb331db-b6cd-4bce-a099-f918904eec77-kube-api-access-8bdbd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4xmrn\" (UID: \"5fb331db-b6cd-4bce-a099-f918904eec77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.157203 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.157115 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:47.589213 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:49:47.589171 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb331db_b6cd_4bce_a099_f918904eec77.slice/crio-93081270e0f638cfead8afaacbc00442f0fb2dfd91e9a8271ffc7c41bf3fb4b9 WatchSource:0}: Error finding container 93081270e0f638cfead8afaacbc00442f0fb2dfd91e9a8271ffc7c41bf3fb4b9: Status 404 returned error can't find the container with id 93081270e0f638cfead8afaacbc00442f0fb2dfd91e9a8271ffc7c41bf3fb4b9 Nov 03 02:49:47.590757 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.590728 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn"] Nov 03 02:49:47.591447 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.591416 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:47.591498 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.591476 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:47.591530 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.591507 4992 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 02:49:47.888373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.888277 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" event={"ID":"5fb331db-b6cd-4bce-a099-f918904eec77","Type":"ContainerStarted","Data":"c41619d8942faf0edacf1b26a3b329493b7cd79079548d098ac0491706dd6c8d"} Nov 03 02:49:47.888373 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:47.888345 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" event={"ID":"5fb331db-b6cd-4bce-a099-f918904eec77","Type":"ContainerStarted","Data":"93081270e0f638cfead8afaacbc00442f0fb2dfd91e9a8271ffc7c41bf3fb4b9"} Nov 03 02:49:48.017882 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:48.017826 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podStartSLOduration=2.01780956 podStartE2EDuration="2.01780956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:48.016781879 +0000 UTC m=+5523.595604465" watchObservedRunningTime="2026-11-03 02:49:48.01780956 +0000 UTC m=+5523.596632170" Nov 03 02:49:48.157851 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:48.157775 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:48.159547 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:48.159519 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:48.159707 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:48.159594 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:49.158180 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:49.158142 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:49.158632 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:49.158219 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:50.158385 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:50.158351 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:50.158795 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:50.158404 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:51.158627 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:51.158584 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:51.159058 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:51.158647 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:52.157633 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:52.157593 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:52.157852 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:52.157656 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:53.158444 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:53.158403 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:53.158886 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:53.158476 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:54.158433 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:54.158387 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Nov 03 02:49:54.158615 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:54.158478 4992 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Nov 03 02:49:55.957468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:55.957431 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:55.958045 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:55.957810 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:49:55.958997 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:49:55.958950 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" Nov 03 02:50:02.150222 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:50:02.150184 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:50:32.154365 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:50:32.154319 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:51:02.156416 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:51:02.156377 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:51:32.149588 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:51:32.149532 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:51:33.269344 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.269306 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7"] Nov 03 02:51:33.272897 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.272878 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.327467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.327435 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:51:33.327467 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.327462 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:51:33.327864 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.327845 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7"] Nov 03 02:51:33.329000 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.328968 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.329116 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.329023 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4gv\" (UniqueName: \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.329158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.329122 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.377388 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.377352 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:51:33.377589 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.377451 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 02:51:33.430076 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.430034 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.430263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.430085 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4gv\" (UniqueName: \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.430263 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.430115 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.430759 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.430733 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.432511 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.432495 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.493710 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.493674 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4gv\" (UniqueName: \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\") pod \"maas-api-revoke-sub-1d76a634-d55x7\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.585825 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.585707 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:33.954638 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:51:33.954603 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d2693a_213e_4c17_9488_750a647d5326.slice/crio-568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd WatchSource:0}: Error finding container 568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd: Status 404 returned error can't find the container with id 568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd Nov 03 02:51:33.956488 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:33.956434 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7"] Nov 03 02:51:34.499087 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:34.499042 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" event={"ID":"28d2693a-213e-4c17-9488-750a647d5326","Type":"ContainerStarted","Data":"0c0156fbd5f388fa8f9690a86a89aa69dd9988d6b9ce1775f45b1972d1f3c425"} Nov 03 02:51:34.499525 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:34.499097 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" event={"ID":"28d2693a-213e-4c17-9488-750a647d5326","Type":"ContainerStarted","Data":"568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd"} Nov 03 02:51:34.618764 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:34.618639 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" podStartSLOduration=1.618620495 podStartE2EDuration="1.618620495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:34.618412006 +0000 UTC m=+5630.197234591" watchObservedRunningTime="2026-11-03 02:51:34.618620495 +0000 UTC m=+5630.197443083" Nov 03 02:51:35.505257 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:35.505220 4992 generic.go:356] "Generic (PLEG): container finished" podID="28d2693a-213e-4c17-9488-750a647d5326" containerID="0c0156fbd5f388fa8f9690a86a89aa69dd9988d6b9ce1775f45b1972d1f3c425" exitCode=0 Nov 03 02:51:35.505692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:35.505270 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" event={"ID":"28d2693a-213e-4c17-9488-750a647d5326","Type":"ContainerDied","Data":"0c0156fbd5f388fa8f9690a86a89aa69dd9988d6b9ce1775f45b1972d1f3c425"} Nov 03 02:51:36.643488 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.643451 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:36.762914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.762836 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\") pod \"28d2693a-213e-4c17-9488-750a647d5326\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " Nov 03 02:51:36.763071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.762938 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\" (UniqueName: \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\") pod \"28d2693a-213e-4c17-9488-750a647d5326\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " Nov 03 02:51:36.763071 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.762967 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\") pod \"28d2693a-213e-4c17-9488-750a647d5326\" (UID: \"28d2693a-213e-4c17-9488-750a647d5326\") " Nov 03 02:51:36.763421 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.763390 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca" pod "28d2693a-213e-4c17-9488-750a647d5326" (UID: "28d2693a-213e-4c17-9488-750a647d5326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:36.765305 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.765237 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d2693a-213e-4c17-9488-750a647d5326-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:36.767601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.765667 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv" pod "28d2693a-213e-4c17-9488-750a647d5326" (UID: "28d2693a-213e-4c17-9488-750a647d5326"). InnerVolumeSpecName "kube-api-access-wr4gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:36.767601 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.765743 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca" pod "28d2693a-213e-4c17-9488-750a647d5326" (UID: "28d2693a-213e-4c17-9488-750a647d5326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:36.865816 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.865774 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d2693a-213e-4c17-9488-750a647d5326-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:36.865816 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:36.865809 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4gv\" (UniqueName: \"kubernetes.io/projected/28d2693a-213e-4c17-9488-750a647d5326-kube-api-access-wr4gv\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:37.516218 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:37.516188 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" Nov 03 02:51:37.516218 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:37.516204 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7" event={"ID":"28d2693a-213e-4c17-9488-750a647d5326","Type":"ContainerDied","Data":"568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd"} Nov 03 02:51:37.516426 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:37.516272 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568c675d09b08dd63ebe8899bb5089f81eb41c5510cb4fa8bfc2064b0dd92ecd" Nov 03 02:51:52.156030 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.155993 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k"] Nov 03 02:51:52.156463 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.156423 4992 cpu_manager.go:414] "containerMap: removing container" podUID="28d2693a-213e-4c17-9488-750a647d5326" containerName="revoke-keys" Nov 03 02:51:52.156463 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.156440 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d2693a-213e-4c17-9488-750a647d5326" containerName="revoke-keys" Nov 03 02:51:52.156550 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.156542 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d2693a-213e-4c17-9488-750a647d5326" containerName="revoke-keys" Nov 03 02:51:52.160013 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.159988 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.214299 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.214236 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k"] Nov 03 02:51:52.264052 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.264011 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 02:51:52.264052 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.264025 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 02:51:52.266055 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.266038 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 02:51:52.266163 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.266152 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 02:51:52.320842 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.320800 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj7xb\" (UniqueName: \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.321040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.320966 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.321040 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.321011 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.421891 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.421852 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.422102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.421901 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.422102 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.422037 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj7xb\" (UniqueName: \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.422598 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.422578 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.424289 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.424265 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.485396 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.485353 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj7xb\" (UniqueName: \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\") pod \"maas-api-revoke-sub-dac8908d-67w8k\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:52.773468 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:52.773378 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:53.143705 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:51:53.143673 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e562a6_9860_4574_b740_c1824347fd32.slice/crio-a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07 WatchSource:0}: Error finding container a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07: Status 404 returned error can't find the container with id a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07 Nov 03 02:51:53.145402 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:53.145376 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k"] Nov 03 02:51:53.599774 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:53.599721 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" event={"ID":"45e562a6-9860-4574-b740-c1824347fd32","Type":"ContainerStarted","Data":"50cb7d71e94aec9b686eb55fc8558c8b6a132c9b5ecad0c297dbbdab020f664f"} Nov 03 02:51:53.600182 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:53.599797 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" event={"ID":"45e562a6-9860-4574-b740-c1824347fd32","Type":"ContainerStarted","Data":"a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07"} Nov 03 02:51:53.721226 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:53.721104 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" podStartSLOduration=1.72108508 podStartE2EDuration="1.72108508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:51:53.721068472 +0000 UTC m=+5649.299891059" watchObservedRunningTime="2026-11-03 02:51:53.72108508 +0000 UTC m=+5649.299907667" Nov 03 02:51:54.642945 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:54.642904 4992 generic.go:356] "Generic (PLEG): container finished" podID="45e562a6-9860-4574-b740-c1824347fd32" containerID="50cb7d71e94aec9b686eb55fc8558c8b6a132c9b5ecad0c297dbbdab020f664f" exitCode=0 Nov 03 02:51:54.643489 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:54.643018 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" event={"ID":"45e562a6-9860-4574-b740-c1824347fd32","Type":"ContainerDied","Data":"50cb7d71e94aec9b686eb55fc8558c8b6a132c9b5ecad0c297dbbdab020f664f"} Nov 03 02:51:56.022787 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.022745 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:56.056543 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.056509 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\" (UniqueName: \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\") pod \"45e562a6-9860-4574-b740-c1824347fd32\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " Nov 03 02:51:56.056739 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.056601 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\") pod \"45e562a6-9860-4574-b740-c1824347fd32\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " Nov 03 02:51:56.056836 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.056627 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\") pod \"45e562a6-9860-4574-b740-c1824347fd32\" (UID: \"45e562a6-9860-4574-b740-c1824347fd32\") " Nov 03 02:51:56.057504 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.057466 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca" pod "45e562a6-9860-4574-b740-c1824347fd32" (UID: "45e562a6-9860-4574-b740-c1824347fd32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 02:51:56.058151 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.058084 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e562a6-9860-4574-b740-c1824347fd32-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:56.074793 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.072688 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca" pod "45e562a6-9860-4574-b740-c1824347fd32" (UID: "45e562a6-9860-4574-b740-c1824347fd32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:51:56.076487 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.076438 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb" pod "45e562a6-9860-4574-b740-c1824347fd32" (UID: "45e562a6-9860-4574-b740-c1824347fd32"). InnerVolumeSpecName "kube-api-access-lj7xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:51:56.159194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.159161 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e562a6-9860-4574-b740-c1824347fd32-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:56.159194 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.159191 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj7xb\" (UniqueName: \"kubernetes.io/projected/45e562a6-9860-4574-b740-c1824347fd32-kube-api-access-lj7xb\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:51:56.655275 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.655241 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" Nov 03 02:51:56.655275 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.655258 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k" event={"ID":"45e562a6-9860-4574-b740-c1824347fd32","Type":"ContainerDied","Data":"a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07"} Nov 03 02:51:56.655552 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:51:56.655320 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a76a95cae2c13d19e73771f308aa8e30d4c8bac3675501714c6d03a5c133fe07" Nov 03 02:52:02.149647 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:52:02.149612 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:52:32.149605 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:52:32.149542 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:52:39.029312 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.029276 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:52:39.029831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.029707 4992 cpu_manager.go:414] "containerMap: removing container" podUID="45e562a6-9860-4574-b740-c1824347fd32" containerName="revoke-keys" Nov 03 02:52:39.029831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.029729 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e562a6-9860-4574-b740-c1824347fd32" containerName="revoke-keys" Nov 03 02:52:39.029831 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.029814 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e562a6-9860-4574-b740-c1824347fd32" containerName="revoke-keys" Nov 03 02:52:39.033484 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.033460 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.087697 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.087664 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:52:39.128692 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128649 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.128914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128702 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.128914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128794 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znkj\" (UniqueName: \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.128914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128850 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.128914 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128895 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.129106 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.128919 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.137358 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.137324 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-65a246\"/\"kube-root-ca.crt\"" Nov 03 02:52:39.137516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.137458 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-65a246\"/\"openshift-service-ca.crt\"" Nov 03 02:52:39.137516 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.137488 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-65a246\"/\"simulator-single-main-65a246-kserve-self-signed-certs\"" Nov 03 02:52:39.139459 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.139444 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-65a246\"/\"default-dockercfg-6lt7x\"" Nov 03 02:52:39.230244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230199 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230244 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230248 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230278 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znkj\" (UniqueName: \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230302 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230331 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230532 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230458 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230734 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230797 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230759 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.230866 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.230815 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.232604 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.232551 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.232921 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.232902 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.293155 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.293081 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znkj\" (UniqueName: \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\") pod \"simulator-single-main-65a246-kserve-66d56599fd-t2w2c\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.344696 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.344658 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:39.720613 ip-10-0-138-19 kubenswrapper[4992]: W1103 02:52:39.720574 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254890dd_8a14_44bf_9d75_ee9e49680efe.slice/crio-0f32f671a2427e6b05e1dc2f4956f62f2c4a7c95321408c3865a8ba8b47be755 WatchSource:0}: Error finding container 0f32f671a2427e6b05e1dc2f4956f62f2c4a7c95321408c3865a8ba8b47be755: Status 404 returned error can't find the container with id 0f32f671a2427e6b05e1dc2f4956f62f2c4a7c95321408c3865a8ba8b47be755 Nov 03 02:52:39.722636 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.722610 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:52:39.900876 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.900841 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" event={"ID":"254890dd-8a14-44bf-9d75-ee9e49680efe","Type":"ContainerStarted","Data":"a35d2f20577b1e20a79c7425288b4c0a778cd90b341a1fabaa9a67d7ba3925c3"} Nov 03 02:52:39.901057 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:39.900885 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" event={"ID":"254890dd-8a14-44bf-9d75-ee9e49680efe","Type":"ContainerStarted","Data":"0f32f671a2427e6b05e1dc2f4956f62f2c4a7c95321408c3865a8ba8b47be755"} Nov 03 02:52:40.022811 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:40.022704 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" podStartSLOduration=2.022689485 podStartE2EDuration="2.022689485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:52:40.021677275 +0000 UTC m=+5695.600499858" watchObservedRunningTime="2026-11-03 02:52:40.022689485 +0000 UTC m=+5695.601512069" Nov 03 02:52:40.344868 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:40.344777 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:45.401928 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:45.401800 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:52:45.412129 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:45.410029 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:52:49.345165 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:49.345130 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:49.357315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:49.357283 4992 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:52:49.971441 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:52:49.971396 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:53:02.150756 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:53:02.150670 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:53:32.150310 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:53:32.150272 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:53:40.057281 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:40.057235 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:53:46.257081 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:46.257048 4992 scope.go:122] "RemoveContainer" containerID="0b7d8f1abebce826d05762a8bd0e46fe762d6211249d2b1151beac02fedeac21" Nov 03 02:53:55.077577 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.077504 4992 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" podUID="254890dd-8a14-44bf-9d75-ee9e49680efe" containerName="main" containerID="cri-o://a35d2f20577b1e20a79c7425288b4c0a778cd90b341a1fabaa9a67d7ba3925c3" gracePeriod=45 Nov 03 02:53:55.289523 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.289492 4992 generic.go:356] "Generic (PLEG): container finished" podID="254890dd-8a14-44bf-9d75-ee9e49680efe" containerID="a35d2f20577b1e20a79c7425288b4c0a778cd90b341a1fabaa9a67d7ba3925c3" exitCode=0 Nov 03 02:53:55.289718 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.289546 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" event={"ID":"254890dd-8a14-44bf-9d75-ee9e49680efe","Type":"ContainerDied","Data":"a35d2f20577b1e20a79c7425288b4c0a778cd90b341a1fabaa9a67d7ba3925c3"} Nov 03 02:53:55.326995 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.326965 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:53:55.370147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370054 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\" (UniqueName: \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.370147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370114 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.370147 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370138 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.370442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370187 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.370442 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370216 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.370553 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370444 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:53:55.370553 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370453 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:53:55.370660 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.370584 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:53:55.372410 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.372386 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 02:53:55.372522 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.372461 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "kube-api-access-2znkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 02:53:55.471158 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471120 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\" (UniqueName: \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\") pod \"254890dd-8a14-44bf-9d75-ee9e49680efe\" (UID: \"254890dd-8a14-44bf-9d75-ee9e49680efe\") " Nov 03 02:53:55.471368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471343 4992 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-home\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:55.471368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471355 4992 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-model-cache\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:55.471368 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471364 4992 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-dshm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:55.471539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471373 4992 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254890dd-8a14-44bf-9d75-ee9e49680efe-tmp-dir\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:55.471539 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.471381 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znkj\" (UniqueName: \"kubernetes.io/projected/254890dd-8a14-44bf-9d75-ee9e49680efe-kube-api-access-2znkj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:55.473237 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.473210 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs" pod "254890dd-8a14-44bf-9d75-ee9e49680efe" (UID: "254890dd-8a14-44bf-9d75-ee9e49680efe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 02:53:55.572771 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:55.572726 4992 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254890dd-8a14-44bf-9d75-ee9e49680efe-tls-certs\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 02:53:56.295844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.295814 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" Nov 03 02:53:56.295844 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.295831 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c" event={"ID":"254890dd-8a14-44bf-9d75-ee9e49680efe","Type":"ContainerDied","Data":"0f32f671a2427e6b05e1dc2f4956f62f2c4a7c95321408c3865a8ba8b47be755"} Nov 03 02:53:56.296374 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.295879 4992 scope.go:122] "RemoveContainer" containerID="a35d2f20577b1e20a79c7425288b4c0a778cd90b341a1fabaa9a67d7ba3925c3" Nov 03 02:53:56.481078 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.481043 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:53:56.532738 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.532699 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-65a246/simulator-single-main-65a246-kserve-66d56599fd-t2w2c"] Nov 03 02:53:56.593923 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:53:56.593888 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254890dd-8a14-44bf-9d75-ee9e49680efe" path="/var/lib/kubelet/pods/254890dd-8a14-44bf-9d75-ee9e49680efe/volumes" Nov 03 02:54:02.151281 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:54:02.151243 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:54:32.150088 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:54:32.150052 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:55:02.149581 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:55:02.149526 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:55:32.150496 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:55:32.150458 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:55:59.148798 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:55:59.148701 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:55:59.148798 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:55:59.148769 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:56:02.149944 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:56:02.149906 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:56:12.002584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:12.001706 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 02:56:12.002584 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:12.001795 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 02:56:32.150609 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:56:32.150568 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:56:37.050136 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:37.050097 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7"] Nov 03 02:56:37.101576 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:37.101533 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d76a634-d55x7"] Nov 03 02:56:38.593990 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:38.593954 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d2693a-213e-4c17-9488-750a647d5326" path="/var/lib/kubelet/pods/28d2693a-213e-4c17-9488-750a647d5326/volumes" Nov 03 02:56:56.051201 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:56.051162 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k"] Nov 03 02:56:56.103000 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:56.102964 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac8908d-67w8k"] Nov 03 02:56:56.593591 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:56:56.593539 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e562a6-9860-4574-b740-c1824347fd32" path="/var/lib/kubelet/pods/45e562a6-9860-4574-b740-c1824347fd32/volumes" Nov 03 02:57:02.150570 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:57:02.150522 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:57:32.149741 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:57:32.149681 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:57:45.435925 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:57:45.435898 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:57:45.444315 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:57:45.444288 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 02:57:46.307159 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:57:46.307129 4992 scope.go:122] "RemoveContainer" containerID="0c0156fbd5f388fa8f9690a86a89aa69dd9988d6b9ce1775f45b1972d1f3c425" Nov 03 02:58:02.150662 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:58:02.150622 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:58:32.149636 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:58:32.149595 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:58:46.327982 ip-10-0-138-19 kubenswrapper[4992]: I1103 02:58:46.327948 4992 scope.go:122] "RemoveContainer" containerID="50cb7d71e94aec9b686eb55fc8558c8b6a132c9b5ecad0c297dbbdab020f664f" Nov 03 02:59:02.165304 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:59:02.165106 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 02:59:32.149488 ip-10-0-138-19 kubenswrapper[4992]: E1103 02:59:32.149452 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:00:00.155050 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.155006 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:00.155505 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.155465 4992 cpu_manager.go:414] "containerMap: removing container" podUID="254890dd-8a14-44bf-9d75-ee9e49680efe" containerName="main" Nov 03 03:00:00.155505 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.155477 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254890dd-8a14-44bf-9d75-ee9e49680efe" containerName="main" Nov 03 03:00:00.155649 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.155543 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="254890dd-8a14-44bf-9d75-ee9e49680efe" containerName="main" Nov 03 03:00:00.158738 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.158710 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.211799 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.211751 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:00.215435 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.215388 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsx7\" (UniqueName: \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.215435 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.215428 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.261795 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.261747 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:00:00.262019 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.261842 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:00:00.262162 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.262141 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:00:00.316426 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.316382 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsx7\" (UniqueName: \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.316426 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.316429 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.317062 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.317041 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.379398 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.379358 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsx7\" (UniqueName: \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\") pod \"maas-api-key-cleanup-29894580-9zjxw\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.472547 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.472502 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:00.844360 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:00:00.844297 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec01dff_5cf4_4f37_bc7b_6e7ca14bf536.slice/crio-dde5b890f9a8e95e186dc4aae6fd3497b4e1f8182aa50685d75fdb73f50a38ba WatchSource:0}: Error finding container dde5b890f9a8e95e186dc4aae6fd3497b4e1f8182aa50685d75fdb73f50a38ba: Status 404 returned error can't find the container with id dde5b890f9a8e95e186dc4aae6fd3497b4e1f8182aa50685d75fdb73f50a38ba Nov 03 03:00:00.847140 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:00.847111 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:01.344544 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:01.344502 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerStarted","Data":"be2f8f1117e83eb15288cf13da50cb4fa81c5b39ad5a71a384f7006eae02ddb4"} Nov 03 03:00:01.344991 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:01.344598 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerStarted","Data":"dde5b890f9a8e95e186dc4aae6fd3497b4e1f8182aa50685d75fdb73f50a38ba"} Nov 03 03:00:01.464381 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:01.464267 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" podStartSLOduration=1.4642504459999999 podStartE2EDuration="1.464250446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:00:01.463954123 +0000 UTC m=+6137.042776711" watchObservedRunningTime="2026-11-03 03:00:01.464250446 +0000 UTC m=+6137.043073030" Nov 03 03:00:02.151634 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:00:02.151528 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:00:02.350703 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:02.350667 4992 generic.go:356] "Generic (PLEG): container finished" podID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerID="be2f8f1117e83eb15288cf13da50cb4fa81c5b39ad5a71a384f7006eae02ddb4" exitCode=60 Nov 03 03:00:02.351185 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:02.350763 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerDied","Data":"be2f8f1117e83eb15288cf13da50cb4fa81c5b39ad5a71a384f7006eae02ddb4"} Nov 03 03:00:02.351185 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:02.350995 4992 scope.go:122] "RemoveContainer" containerID="be2f8f1117e83eb15288cf13da50cb4fa81c5b39ad5a71a384f7006eae02ddb4" Nov 03 03:00:03.357650 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:03.357612 4992 generic.go:356] "Generic (PLEG): container finished" podID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerID="0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71" exitCode=60 Nov 03 03:00:03.358131 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:03.357711 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerDied","Data":"0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71"} Nov 03 03:00:03.358131 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:03.357776 4992 scope.go:122] "RemoveContainer" containerID="be2f8f1117e83eb15288cf13da50cb4fa81c5b39ad5a71a384f7006eae02ddb4" Nov 03 03:00:03.358131 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:03.357957 4992 scope.go:122] "RemoveContainer" containerID="0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71" Nov 03 03:00:03.358252 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:00:03.358219 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-9zjxw_odh-ai-gateway-infra(8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" Nov 03 03:00:04.366781 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:04.366738 4992 scope.go:122] "RemoveContainer" containerID="0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71" Nov 03 03:00:04.367200 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:00:04.366968 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-9zjxw_odh-ai-gateway-infra(8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" Nov 03 03:00:13.587502 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:13.587468 4992 scope.go:122] "RemoveContainer" containerID="0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71" Nov 03 03:00:14.419077 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:14.419043 4992 generic.go:356] "Generic (PLEG): container finished" podID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerID="bd418ee3e2ba8a3c7562f8b010f2e7c9c1686fe0342df99e586749a373b473d0" exitCode=60 Nov 03 03:00:14.419260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:14.419112 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerDied","Data":"bd418ee3e2ba8a3c7562f8b010f2e7c9c1686fe0342df99e586749a373b473d0"} Nov 03 03:00:14.419260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:14.419174 4992 scope.go:122] "RemoveContainer" containerID="0d4d34120f3fd73419feb7cd77a2fac5770606401e6faf41a65a82d1a5c28c71" Nov 03 03:00:14.419513 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:14.419496 4992 scope.go:122] "RemoveContainer" containerID="bd418ee3e2ba8a3c7562f8b010f2e7c9c1686fe0342df99e586749a373b473d0" Nov 03 03:00:14.419803 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:00:14.419778 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894580-9zjxw_odh-ai-gateway-infra(8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" Nov 03 03:00:15.554891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.554847 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:15.700523 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.700492 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:15.877122 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.877036 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\") pod \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " Nov 03 03:00:15.877287 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.877187 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\" (UniqueName: \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\") pod \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\" (UID: \"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536\") " Nov 03 03:00:15.877553 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.877521 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca" pod "8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" (UID: "8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:00:15.879281 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.879258 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7" pod "8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" (UID: "8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536"). InnerVolumeSpecName "kube-api-access-nhsx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:00:15.978825 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.978788 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:15.978825 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:15.978822 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsx7\" (UniqueName: \"kubernetes.io/projected/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536-kube-api-access-nhsx7\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:00:16.433014 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:16.432976 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" Nov 03 03:00:16.433014 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:16.432992 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw" event={"ID":"8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536","Type":"ContainerDied","Data":"dde5b890f9a8e95e186dc4aae6fd3497b4e1f8182aa50685d75fdb73f50a38ba"} Nov 03 03:00:16.433313 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:16.433054 4992 scope.go:122] "RemoveContainer" containerID="bd418ee3e2ba8a3c7562f8b010f2e7c9c1686fe0342df99e586749a373b473d0" Nov 03 03:00:16.605353 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:16.605314 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:16.657334 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:16.657252 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894580-9zjxw"] Nov 03 03:00:18.634674 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:00:18.619419 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" path="/var/lib/kubelet/pods/8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536/volumes" Nov 03 03:00:32.150941 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:00:32.150893 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:01:02.150537 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:01:02.150498 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:01:32.149905 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:01:32.149866 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:02:02.151822 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:02:02.151735 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:02:32.151343 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:02:32.151300 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:02:45.549188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:02:45.549067 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:02:45.553269 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:02:45.552945 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:03:02.152346 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:03:02.152302 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:03:32.152573 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:03:32.152520 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:04:02.151351 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:04:02.151308 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:04:32.151972 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:04:32.151931 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:05:02.151047 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:05:02.150959 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:05:32.151593 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:05:32.151538 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:05:32.624174 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624129 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8"] Nov 03 03:05:32.624679 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624657 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.624802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624680 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.624802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624741 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.624802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624751 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.625125 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624855 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.625125 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624871 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.625125 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.624885 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:05:32.630411 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.629651 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.665048 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.665008 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.665272 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.665184 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.665272 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.665254 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d856r\" (UniqueName: \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.726599 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.726542 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:05:32.726810 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.726648 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:05:32.726810 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.726549 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:05:32.728591 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.728546 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8"] Nov 03 03:05:32.731855 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.731825 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:05:32.766464 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.766422 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.766707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.766471 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d856r\" (UniqueName: \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.766707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.766603 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.767274 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.767247 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.769081 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.769049 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.830278 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.830239 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d856r\" (UniqueName: \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\") pod \"maas-api-revoke-sub-802867b1-8gmn8\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:32.946476 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:32.946434 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:33.324840 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:05:33.324803 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4136812f_1802_4492_80a6_a4d6704f9edd.slice/crio-a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d WatchSource:0}: Error finding container a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d: Status 404 returned error can't find the container with id a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d Nov 03 03:05:33.326453 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:33.326324 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8"] Nov 03 03:05:34.214825 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:34.214792 4992 generic.go:356] "Generic (PLEG): container finished" podID="4136812f-1802-4492-80a6-a4d6704f9edd" containerID="a0a570f920cad2f5990a09360a9b3941cc754f5d6db3e22d69ed71c611343452" exitCode=0 Nov 03 03:05:34.215020 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:34.214877 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" event={"ID":"4136812f-1802-4492-80a6-a4d6704f9edd","Type":"ContainerDied","Data":"a0a570f920cad2f5990a09360a9b3941cc754f5d6db3e22d69ed71c611343452"} Nov 03 03:05:34.215020 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:34.214931 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" event={"ID":"4136812f-1802-4492-80a6-a4d6704f9edd","Type":"ContainerStarted","Data":"a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d"} Nov 03 03:05:35.360069 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.360038 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:35.394712 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.394652 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\" (UniqueName: \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\") pod \"4136812f-1802-4492-80a6-a4d6704f9edd\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " Nov 03 03:05:35.394974 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.394813 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\") pod \"4136812f-1802-4492-80a6-a4d6704f9edd\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " Nov 03 03:05:35.394974 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.394910 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\") pod \"4136812f-1802-4492-80a6-a4d6704f9edd\" (UID: \"4136812f-1802-4492-80a6-a4d6704f9edd\") " Nov 03 03:05:35.395181 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.395155 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca" pod "4136812f-1802-4492-80a6-a4d6704f9edd" (UID: "4136812f-1802-4492-80a6-a4d6704f9edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:05:35.396865 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.396840 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca" pod "4136812f-1802-4492-80a6-a4d6704f9edd" (UID: "4136812f-1802-4492-80a6-a4d6704f9edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:05:35.396990 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.396977 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r" pod "4136812f-1802-4492-80a6-a4d6704f9edd" (UID: "4136812f-1802-4492-80a6-a4d6704f9edd"). InnerVolumeSpecName "kube-api-access-d856r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:05:35.496306 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.496204 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d856r\" (UniqueName: \"kubernetes.io/projected/4136812f-1802-4492-80a6-a4d6704f9edd-kube-api-access-d856r\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:35.496306 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.496256 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:35.496306 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:35.496273 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4136812f-1802-4492-80a6-a4d6704f9edd-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:05:36.226645 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:36.226614 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" Nov 03 03:05:36.226830 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:36.226610 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8" event={"ID":"4136812f-1802-4492-80a6-a4d6704f9edd","Type":"ContainerDied","Data":"a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d"} Nov 03 03:05:36.226830 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:05:36.226742 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a2bf73b04ce79219c3c1dc15c8ab53c483fbc205a03146e74ed786c2fd3b8d" Nov 03 03:06:02.154663 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:06:02.154621 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:06:32.151300 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:06:32.151252 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:07:02.150982 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:07:02.150944 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:07:32.151403 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:07:32.151361 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:07:45.580978 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:07:45.580851 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:07:45.619539 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:07:45.619509 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:08:02.151436 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:08:02.151348 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:08:32.151408 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:08:32.151364 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:09:02.152480 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:09:02.152435 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:09:23.734330 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.734285 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2"] Nov 03 03:09:23.735008 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.734953 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:09:23.735008 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.734979 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec01dff-5cf4-4f37-bc7b-6e7ca14bf536" containerName="cleanup" Nov 03 03:09:23.735143 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.735017 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4136812f-1802-4492-80a6-a4d6704f9edd" containerName="revoke-keys" Nov 03 03:09:23.735143 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.735025 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4136812f-1802-4492-80a6-a4d6704f9edd" containerName="revoke-keys" Nov 03 03:09:23.735263 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.735155 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4136812f-1802-4492-80a6-a4d6704f9edd" containerName="revoke-keys" Nov 03 03:09:23.738461 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.738435 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.785226 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.785129 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtm9v\" (UniqueName: \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.785439 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.785354 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.785507 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.785468 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.792335 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.792301 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2"] Nov 03 03:09:23.842345 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.842304 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:09:23.842345 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.842303 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:09:23.844681 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.844646 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:09:23.844681 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.844657 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:09:23.886608 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.886542 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.886608 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.886618 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.886829 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.886650 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtm9v\" (UniqueName: \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.887405 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.887381 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.889036 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.889016 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:23.950423 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:23.950382 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtm9v\" (UniqueName: \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\") pod \"maas-api-revoke-sub-84687dc3-d6ql2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:24.052247 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:24.052148 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:24.424622 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:09:24.424586 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f02395_bb1e_4f09_bc3a_2669c96805f2.slice/crio-17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d WatchSource:0}: Error finding container 17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d: Status 404 returned error can't find the container with id 17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d Nov 03 03:09:24.425887 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:24.425863 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2"] Nov 03 03:09:24.480638 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:24.480517 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" event={"ID":"d2f02395-bb1e-4f09-bc3a-2669c96805f2","Type":"ContainerStarted","Data":"17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d"} Nov 03 03:09:25.488054 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:25.487959 4992 generic.go:356] "Generic (PLEG): container finished" podID="d2f02395-bb1e-4f09-bc3a-2669c96805f2" containerID="88512cc98fce61765a880640af064be4ef3759fa25fce3726417dbe69ff286d8" exitCode=0 Nov 03 03:09:25.488054 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:25.488042 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" event={"ID":"d2f02395-bb1e-4f09-bc3a-2669c96805f2","Type":"ContainerDied","Data":"88512cc98fce61765a880640af064be4ef3759fa25fce3726417dbe69ff286d8"} Nov 03 03:09:26.633447 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.633413 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:26.713904 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.713864 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\") pod \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " Nov 03 03:09:26.714123 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.713918 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\") pod \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " Nov 03 03:09:26.714123 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.713950 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\" (UniqueName: \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\") pod \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\" (UID: \"d2f02395-bb1e-4f09-bc3a-2669c96805f2\") " Nov 03 03:09:26.714355 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.714331 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca" pod "d2f02395-bb1e-4f09-bc3a-2669c96805f2" (UID: "d2f02395-bb1e-4f09-bc3a-2669c96805f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:26.716158 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.716120 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca" pod "d2f02395-bb1e-4f09-bc3a-2669c96805f2" (UID: "d2f02395-bb1e-4f09-bc3a-2669c96805f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:09:26.716365 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.716344 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v" pod "d2f02395-bb1e-4f09-bc3a-2669c96805f2" (UID: "d2f02395-bb1e-4f09-bc3a-2669c96805f2"). InnerVolumeSpecName "kube-api-access-mtm9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:26.815092 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.814978 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:26.815092 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.815033 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f02395-bb1e-4f09-bc3a-2669c96805f2-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:26.815092 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:26.815045 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtm9v\" (UniqueName: \"kubernetes.io/projected/d2f02395-bb1e-4f09-bc3a-2669c96805f2-kube-api-access-mtm9v\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:27.500071 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:27.500032 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" event={"ID":"d2f02395-bb1e-4f09-bc3a-2669c96805f2","Type":"ContainerDied","Data":"17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d"} Nov 03 03:09:27.500260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:27.500090 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f5949d1fa9a932acf27d5c2a41201164504ace7d07a3c646cfc22443078d2d" Nov 03 03:09:27.500260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:27.500048 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2" Nov 03 03:09:32.151889 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:09:32.151851 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:09:34.557290 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.557244 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q"] Nov 03 03:09:34.557787 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.557769 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d2f02395-bb1e-4f09-bc3a-2669c96805f2" containerName="revoke-keys" Nov 03 03:09:34.557836 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.557788 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f02395-bb1e-4f09-bc3a-2669c96805f2" containerName="revoke-keys" Nov 03 03:09:34.557873 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.557859 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f02395-bb1e-4f09-bc3a-2669c96805f2" containerName="revoke-keys" Nov 03 03:09:34.561435 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.561411 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.615030 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.614995 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q"] Nov 03 03:09:34.664015 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.663974 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:09:34.664015 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.664002 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:09:34.664266 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.664080 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:09:34.667689 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.667663 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:09:34.696247 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.696206 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxj8\" (UniqueName: \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.696343 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.696261 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.696456 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.696430 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.797990 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.797948 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxj8\" (UniqueName: \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.797990 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.797993 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.798227 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.798057 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.798789 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.798762 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.800438 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.800417 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.865625 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.865534 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxj8\" (UniqueName: \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\") pod \"maas-api-revoke-sub-e81de716-wsl5q\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:34.877373 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:34.877335 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:35.252532 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:09:35.252500 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ec39c9_500e_4036_bf0a_b9bf3abf143e.slice/crio-e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee WatchSource:0}: Error finding container e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee: Status 404 returned error can't find the container with id e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee Nov 03 03:09:35.253360 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:35.253320 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q"] Nov 03 03:09:35.542515 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:35.542425 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" event={"ID":"38ec39c9-500e-4036-bf0a-b9bf3abf143e","Type":"ContainerStarted","Data":"e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee"} Nov 03 03:09:36.552201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:36.552108 4992 generic.go:356] "Generic (PLEG): container finished" podID="38ec39c9-500e-4036-bf0a-b9bf3abf143e" containerID="72c41e12cca739bc075eb38ec5836624e3b17a9927d6af905b95d55d7fd3f76c" exitCode=0 Nov 03 03:09:36.552622 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:36.552196 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" event={"ID":"38ec39c9-500e-4036-bf0a-b9bf3abf143e","Type":"ContainerDied","Data":"72c41e12cca739bc075eb38ec5836624e3b17a9927d6af905b95d55d7fd3f76c"} Nov 03 03:09:37.709951 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.709919 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:37.826433 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.826398 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\") pod \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " Nov 03 03:09:37.826808 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.826776 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\") pod \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " Nov 03 03:09:37.826969 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.826824 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\" (UniqueName: \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\") pod \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\" (UID: \"38ec39c9-500e-4036-bf0a-b9bf3abf143e\") " Nov 03 03:09:37.830103 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.830068 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca" pod "38ec39c9-500e-4036-bf0a-b9bf3abf143e" (UID: "38ec39c9-500e-4036-bf0a-b9bf3abf143e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:09:37.830263 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.830170 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8" pod "38ec39c9-500e-4036-bf0a-b9bf3abf143e" (UID: "38ec39c9-500e-4036-bf0a-b9bf3abf143e"). InnerVolumeSpecName "kube-api-access-9jxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:09:37.831895 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.831858 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca" pod "38ec39c9-500e-4036-bf0a-b9bf3abf143e" (UID: "38ec39c9-500e-4036-bf0a-b9bf3abf143e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:09:37.928089 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.928047 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:37.928089 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.928088 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ec39c9-500e-4036-bf0a-b9bf3abf143e-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:37.928318 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:37.928099 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jxj8\" (UniqueName: \"kubernetes.io/projected/38ec39c9-500e-4036-bf0a-b9bf3abf143e-kube-api-access-9jxj8\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:09:38.563995 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:38.563958 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" Nov 03 03:09:38.563995 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:38.563982 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q" event={"ID":"38ec39c9-500e-4036-bf0a-b9bf3abf143e","Type":"ContainerDied","Data":"e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee"} Nov 03 03:09:38.564222 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:09:38.564036 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e613ed50dfb87b209a082b0134738f0c3f65ccc3d25845f8efc0dc794c8aee" Nov 03 03:10:02.150576 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:10:02.150526 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:10:08.800891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.800846 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb"] Nov 03 03:10:08.801425 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.801264 4992 cpu_manager.go:414] "containerMap: removing container" podUID="38ec39c9-500e-4036-bf0a-b9bf3abf143e" containerName="revoke-keys" Nov 03 03:10:08.801425 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.801276 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ec39c9-500e-4036-bf0a-b9bf3abf143e" containerName="revoke-keys" Nov 03 03:10:08.801425 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.801368 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ec39c9-500e-4036-bf0a-b9bf3abf143e" containerName="revoke-keys" Nov 03 03:10:08.804772 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.804750 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.847009 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.846969 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.847190 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.847038 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.847190 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.847153 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvr4\" (UniqueName: \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.864758 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.864719 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb"] Nov 03 03:10:08.913962 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.913927 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:10:08.914236 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.914216 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:10:08.914282 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.914222 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:10:08.914319 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.914282 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:10:08.947751 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.947712 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvr4\" (UniqueName: \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.947934 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.947810 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.947934 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.947873 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.948607 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.948576 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:08.950381 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:08.950350 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:09.011656 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:09.011610 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvr4\" (UniqueName: \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\") pod \"maas-api-revoke-sub-8fba94b6-krpjb\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:09.119648 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:09.119512 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:09.492208 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:10:09.492174 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97d974a_4175_4db6_b53f_697e0eb685a9.slice/crio-8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019 WatchSource:0}: Error finding container 8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019: Status 404 returned error can't find the container with id 8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019 Nov 03 03:10:09.493803 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:09.493776 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb"] Nov 03 03:10:09.726801 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:09.726763 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" event={"ID":"e97d974a-4175-4db6-b53f-697e0eb685a9","Type":"ContainerStarted","Data":"8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019"} Nov 03 03:10:10.733809 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:10.733773 4992 generic.go:356] "Generic (PLEG): container finished" podID="e97d974a-4175-4db6-b53f-697e0eb685a9" containerID="209da674543c88b6027207ec6e77efdbb792f1644ed2eed4129a8a02e07db8f6" exitCode=0 Nov 03 03:10:10.734238 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:10.733836 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" event={"ID":"e97d974a-4175-4db6-b53f-697e0eb685a9","Type":"ContainerDied","Data":"209da674543c88b6027207ec6e77efdbb792f1644ed2eed4129a8a02e07db8f6"} Nov 03 03:10:11.881258 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.881230 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:11.974857 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.974816 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\") pod \"e97d974a-4175-4db6-b53f-697e0eb685a9\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " Nov 03 03:10:11.975048 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.974892 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\" (UniqueName: \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\") pod \"e97d974a-4175-4db6-b53f-697e0eb685a9\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " Nov 03 03:10:11.975048 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.974966 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\") pod \"e97d974a-4175-4db6-b53f-697e0eb685a9\" (UID: \"e97d974a-4175-4db6-b53f-697e0eb685a9\") " Nov 03 03:10:11.975231 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.975206 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca" pod "e97d974a-4175-4db6-b53f-697e0eb685a9" (UID: "e97d974a-4175-4db6-b53f-697e0eb685a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:10:11.977033 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.977006 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4" pod "e97d974a-4175-4db6-b53f-697e0eb685a9" (UID: "e97d974a-4175-4db6-b53f-697e0eb685a9"). InnerVolumeSpecName "kube-api-access-7rvr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:11.977033 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:11.977020 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca" pod "e97d974a-4175-4db6-b53f-697e0eb685a9" (UID: "e97d974a-4175-4db6-b53f-697e0eb685a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:12.076621 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.076525 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:12.076621 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.076589 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvr4\" (UniqueName: \"kubernetes.io/projected/e97d974a-4175-4db6-b53f-697e0eb685a9-kube-api-access-7rvr4\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:12.076621 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.076600 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97d974a-4175-4db6-b53f-697e0eb685a9-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:12.745676 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.745638 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" Nov 03 03:10:12.745676 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.745649 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb" event={"ID":"e97d974a-4175-4db6-b53f-697e0eb685a9","Type":"ContainerDied","Data":"8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019"} Nov 03 03:10:12.745964 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:12.745710 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e7d75844d8972ea3f2558746e19ad3ebe1922ee8aed0448f09a257e2cb93019" Nov 03 03:10:32.152587 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:10:32.152533 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:10:36.088514 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:36.088464 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8"] Nov 03 03:10:36.138635 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:36.138246 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-802867b1-8gmn8"] Nov 03 03:10:36.631685 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:36.631642 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4136812f-1802-4492-80a6-a4d6704f9edd" path="/var/lib/kubelet/pods/4136812f-1802-4492-80a6-a4d6704f9edd/volumes" Nov 03 03:10:52.102585 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.102523 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs"] Nov 03 03:10:52.103135 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.102986 4992 cpu_manager.go:414] "containerMap: removing container" podUID="e97d974a-4175-4db6-b53f-697e0eb685a9" containerName="revoke-keys" Nov 03 03:10:52.103135 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.103000 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97d974a-4175-4db6-b53f-697e0eb685a9" containerName="revoke-keys" Nov 03 03:10:52.103135 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.103124 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97d974a-4175-4db6-b53f-697e0eb685a9" containerName="revoke-keys" Nov 03 03:10:52.106777 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.106756 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.159828 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.159790 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs"] Nov 03 03:10:52.204339 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.204298 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.204573 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.204362 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.204573 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.204487 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslq6\" (UniqueName: \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.209370 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.209334 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:10:52.209552 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.209415 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:10:52.209552 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.209398 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:10:52.212957 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.212930 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:10:52.306126 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.306075 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.306126 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.306133 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.306365 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.306194 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslq6\" (UniqueName: \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.307089 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.307065 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.308730 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.308702 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.369973 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.369890 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslq6\" (UniqueName: \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\") pod \"maas-api-revoke-sub-517d35eb-lmlzs\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.420634 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.420555 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:52.793706 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:10:52.793662 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab9279c_6710_4cc0_9294_179dec61a3cf.slice/crio-1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6 WatchSource:0}: Error finding container 1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6: Status 404 returned error can't find the container with id 1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6 Nov 03 03:10:52.795701 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.795665 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs"] Nov 03 03:10:52.994125 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:52.994090 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" event={"ID":"2ab9279c-6710-4cc0-9294-179dec61a3cf","Type":"ContainerStarted","Data":"1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6"} Nov 03 03:10:54.000926 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:54.000882 4992 generic.go:356] "Generic (PLEG): container finished" podID="2ab9279c-6710-4cc0-9294-179dec61a3cf" containerID="5595bbc3423c2fa337f4226420a40e51f4b4a2e8f189fd8a3ff1de618f72f546" exitCode=0 Nov 03 03:10:54.001456 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:54.001018 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" event={"ID":"2ab9279c-6710-4cc0-9294-179dec61a3cf","Type":"ContainerDied","Data":"5595bbc3423c2fa337f4226420a40e51f4b4a2e8f189fd8a3ff1de618f72f546"} Nov 03 03:10:55.151502 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.151460 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:55.240093 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.240046 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\" (UniqueName: \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\") pod \"2ab9279c-6710-4cc0-9294-179dec61a3cf\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " Nov 03 03:10:55.240093 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.240098 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\") pod \"2ab9279c-6710-4cc0-9294-179dec61a3cf\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " Nov 03 03:10:55.240354 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.240219 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\") pod \"2ab9279c-6710-4cc0-9294-179dec61a3cf\" (UID: \"2ab9279c-6710-4cc0-9294-179dec61a3cf\") " Nov 03 03:10:55.240536 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.240511 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca" pod "2ab9279c-6710-4cc0-9294-179dec61a3cf" (UID: "2ab9279c-6710-4cc0-9294-179dec61a3cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:10:55.242400 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.242361 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca" pod "2ab9279c-6710-4cc0-9294-179dec61a3cf" (UID: "2ab9279c-6710-4cc0-9294-179dec61a3cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:10:55.242400 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.242388 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6" pod "2ab9279c-6710-4cc0-9294-179dec61a3cf" (UID: "2ab9279c-6710-4cc0-9294-179dec61a3cf"). InnerVolumeSpecName "kube-api-access-qslq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:10:55.341392 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.341293 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:55.341392 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.341332 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslq6\" (UniqueName: \"kubernetes.io/projected/2ab9279c-6710-4cc0-9294-179dec61a3cf-kube-api-access-qslq6\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:55.341392 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:55.341347 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab9279c-6710-4cc0-9294-179dec61a3cf-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:10:56.013770 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:56.013735 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" event={"ID":"2ab9279c-6710-4cc0-9294-179dec61a3cf","Type":"ContainerDied","Data":"1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6"} Nov 03 03:10:56.013962 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:56.013779 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs" Nov 03 03:10:56.013962 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:10:56.013788 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd528540ba1de728861b96b9ee70f807d33c37f4622d102306977ad3869bed6" Nov 03 03:11:02.151289 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:11:02.151245 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:11:02.915207 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.915166 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c"] Nov 03 03:11:02.915628 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.915613 4992 cpu_manager.go:414] "containerMap: removing container" podUID="2ab9279c-6710-4cc0-9294-179dec61a3cf" containerName="revoke-keys" Nov 03 03:11:02.915688 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.915629 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab9279c-6710-4cc0-9294-179dec61a3cf" containerName="revoke-keys" Nov 03 03:11:02.915726 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.915717 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab9279c-6710-4cc0-9294-179dec61a3cf" containerName="revoke-keys" Nov 03 03:11:02.919108 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.919090 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:02.973937 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:02.973890 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c"] Nov 03 03:11:03.017456 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.017421 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.017696 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.017489 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskjl\" (UniqueName: \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.017696 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.017609 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.023249 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.023213 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:03.023249 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.023236 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:03.023249 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.023211 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:03.023487 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.023262 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:11:03.118224 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.118168 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskjl\" (UniqueName: \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.118224 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.118232 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.118474 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.118345 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.119044 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.119022 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.120644 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.120618 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.181707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.181622 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskjl\" (UniqueName: \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\") pod \"maas-api-revoke-sub-73755308-rgl9c\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.231749 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.231693 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:03.607787 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:03.607758 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c"] Nov 03 03:11:04.056624 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:04.056552 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" event={"ID":"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8","Type":"ContainerStarted","Data":"f873b6266d7b50e5ab2c9e662329ae7c8ddc3078592f816b8b9d7fb0c1f7d23a"} Nov 03 03:11:04.056837 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:04.056645 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" event={"ID":"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8","Type":"ContainerStarted","Data":"622ae260dc6c62f3e55f9de158473b46fb24c00c9bb97fa34120af5513b26fda"} Nov 03 03:11:04.230267 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:04.230196 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" podStartSLOduration=2.230177781 podStartE2EDuration="2.230177781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:11:04.176626407 +0000 UTC m=+6799.755448996" watchObservedRunningTime="2026-11-03 03:11:04.230177781 +0000 UTC m=+6799.809000368" Nov 03 03:11:05.063325 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:05.063292 4992 generic.go:356] "Generic (PLEG): container finished" podID="1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" containerID="f873b6266d7b50e5ab2c9e662329ae7c8ddc3078592f816b8b9d7fb0c1f7d23a" exitCode=0 Nov 03 03:11:05.063529 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:05.063379 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" event={"ID":"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8","Type":"ContainerDied","Data":"f873b6266d7b50e5ab2c9e662329ae7c8ddc3078592f816b8b9d7fb0c1f7d23a"} Nov 03 03:11:06.214594 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.214552 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:06.246721 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.246680 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\") pod \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " Nov 03 03:11:06.246891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.246731 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\" (UniqueName: \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\") pod \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " Nov 03 03:11:06.246891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.246794 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\") pod \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\" (UID: \"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8\") " Nov 03 03:11:06.247102 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.247076 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca" pod "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" (UID: "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:06.248924 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.248884 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca" pod "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" (UID: "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:06.249016 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.248950 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl" pod "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" (UID: "1e3a064f-7938-43a3-9b0e-4b95b57fdcc8"). InnerVolumeSpecName "kube-api-access-tskjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:06.347582 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.347477 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:06.347582 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.347514 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:06.347582 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:06.347526 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskjl\" (UniqueName: \"kubernetes.io/projected/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8-kube-api-access-tskjl\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:07.075391 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:07.075352 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" Nov 03 03:11:07.075594 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:07.075362 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c" event={"ID":"1e3a064f-7938-43a3-9b0e-4b95b57fdcc8","Type":"ContainerDied","Data":"622ae260dc6c62f3e55f9de158473b46fb24c00c9bb97fa34120af5513b26fda"} Nov 03 03:11:07.075594 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:07.075531 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="622ae260dc6c62f3e55f9de158473b46fb24c00c9bb97fa34120af5513b26fda" Nov 03 03:11:32.150850 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:11:32.150807 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:11:41.999434 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:41.998784 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4xmrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:11:41.999434 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:41.998881 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4xmrn" podUID="5fb331db-b6cd-4bce-a099-f918904eec77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:11:46.441972 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:46.441937 4992 scope.go:122] "RemoveContainer" containerID="a0a570f920cad2f5990a09360a9b3941cc754f5d6db3e22d69ed71c611343452" Nov 03 03:11:49.692612 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.692569 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m"] Nov 03 03:11:49.693220 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.693196 4992 cpu_manager.go:414] "containerMap: removing container" podUID="1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" containerName="revoke-keys" Nov 03 03:11:49.693311 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.693224 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" containerName="revoke-keys" Nov 03 03:11:49.693370 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.693344 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" containerName="revoke-keys" Nov 03 03:11:49.697927 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.697902 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.750823 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.750781 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m"] Nov 03 03:11:49.800337 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.800298 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:11:49.800539 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.800340 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:11:49.800539 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.800366 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:11:49.804189 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.804160 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:11:49.826586 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.826521 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.826586 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.826573 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.826844 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.826679 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgv8\" (UniqueName: \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.928977 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.928326 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgv8\" (UniqueName: \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.928977 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.928516 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.928977 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.928598 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.930745 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.929893 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.937283 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.937249 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:49.994030 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:49.993923 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgv8\" (UniqueName: \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\") pod \"maas-api-revoke-sub-50e04e8c-qsv2m\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:50.011129 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:50.011092 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:50.383092 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:11:50.383050 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433c37e6_30b5_41c6_a667_0b9eb50c96b9.slice/crio-993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4 WatchSource:0}: Error finding container 993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4: Status 404 returned error can't find the container with id 993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4 Nov 03 03:11:50.384872 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:50.384847 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m"] Nov 03 03:11:51.384317 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:51.384284 4992 generic.go:356] "Generic (PLEG): container finished" podID="433c37e6-30b5-41c6-a667-0b9eb50c96b9" containerID="183dd922068e801e73103698df701c0418a2e4ee5387fc36c2c72ba79e916194" exitCode=0 Nov 03 03:11:51.384808 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:51.384378 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" event={"ID":"433c37e6-30b5-41c6-a667-0b9eb50c96b9","Type":"ContainerDied","Data":"183dd922068e801e73103698df701c0418a2e4ee5387fc36c2c72ba79e916194"} Nov 03 03:11:51.384808 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:51.384438 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" event={"ID":"433c37e6-30b5-41c6-a667-0b9eb50c96b9","Type":"ContainerStarted","Data":"993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4"} Nov 03 03:11:52.535693 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.535664 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:52.666034 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.666008 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\" (UniqueName: \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\") pod \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " Nov 03 03:11:52.666204 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.666069 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\") pod \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " Nov 03 03:11:52.666204 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.666177 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\") pod \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\" (UID: \"433c37e6-30b5-41c6-a667-0b9eb50c96b9\") " Nov 03 03:11:52.666488 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.666459 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca" pod "433c37e6-30b5-41c6-a667-0b9eb50c96b9" (UID: "433c37e6-30b5-41c6-a667-0b9eb50c96b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:11:52.666643 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.666622 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.668329 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.668302 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca" pod "433c37e6-30b5-41c6-a667-0b9eb50c96b9" (UID: "433c37e6-30b5-41c6-a667-0b9eb50c96b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:11:52.668426 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.668385 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8" pod "433c37e6-30b5-41c6-a667-0b9eb50c96b9" (UID: "433c37e6-30b5-41c6-a667-0b9eb50c96b9"). InnerVolumeSpecName "kube-api-access-wpgv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:11:52.768150 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.768105 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgv8\" (UniqueName: \"kubernetes.io/projected/433c37e6-30b5-41c6-a667-0b9eb50c96b9-kube-api-access-wpgv8\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:52.768150 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:52.768144 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433c37e6-30b5-41c6-a667-0b9eb50c96b9-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:11:53.396195 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:53.396159 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" Nov 03 03:11:53.396383 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:53.396196 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m" event={"ID":"433c37e6-30b5-41c6-a667-0b9eb50c96b9","Type":"ContainerDied","Data":"993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4"} Nov 03 03:11:53.396383 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:11:53.396248 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993bbda702004f9caa16b9167854272cab2311b5d1c355f4251ec3ffe88e6ab4" Nov 03 03:12:00.543688 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.543650 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k"] Nov 03 03:12:00.544099 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.544086 4992 cpu_manager.go:414] "containerMap: removing container" podUID="433c37e6-30b5-41c6-a667-0b9eb50c96b9" containerName="revoke-keys" Nov 03 03:12:00.544145 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.544101 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433c37e6-30b5-41c6-a667-0b9eb50c96b9" containerName="revoke-keys" Nov 03 03:12:00.544191 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.544181 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="433c37e6-30b5-41c6-a667-0b9eb50c96b9" containerName="revoke-keys" Nov 03 03:12:00.547628 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.547605 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.601453 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.601371 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k"] Nov 03 03:12:00.649386 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.649341 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.649638 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.649395 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.649638 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.649529 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.650858 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.650832 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:00.650964 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.650869 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:00.650964 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.650904 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:00.653831 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.653811 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:12:00.750846 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.750789 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.751079 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.750860 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.751079 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.750897 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.751710 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.751678 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.753495 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.753467 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.814686 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.814603 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\") pod \"maas-api-revoke-sub-91d8f222-bwg4k\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:00.861655 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:00.861618 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:01.245711 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:12:01.245675 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168d9445_1e4f_4fd6_a05c_0c7ad890d519.slice/crio-12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27 WatchSource:0}: Error finding container 12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27: Status 404 returned error can't find the container with id 12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27 Nov 03 03:12:01.246570 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:01.246520 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k"] Nov 03 03:12:01.442628 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:01.442585 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" event={"ID":"168d9445-1e4f-4fd6-a05c-0c7ad890d519","Type":"ContainerStarted","Data":"12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27"} Nov 03 03:12:02.153521 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:12:02.153482 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:12:02.449153 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:02.449120 4992 generic.go:356] "Generic (PLEG): container finished" podID="168d9445-1e4f-4fd6-a05c-0c7ad890d519" containerID="80394a4c85bbb6d0e68449a066f302e6931e1c09a7a69b97a5407ac34b078fea" exitCode=0 Nov 03 03:12:02.449360 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:02.449213 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" event={"ID":"168d9445-1e4f-4fd6-a05c-0c7ad890d519","Type":"ContainerDied","Data":"80394a4c85bbb6d0e68449a066f302e6931e1c09a7a69b97a5407ac34b078fea"} Nov 03 03:12:03.631789 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.631754 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:03.819646 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.818999 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\") pod \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " Nov 03 03:12:03.819646 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.819097 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\") pod \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " Nov 03 03:12:03.819646 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.819182 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\") pod \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\" (UID: \"168d9445-1e4f-4fd6-a05c-0c7ad890d519\") " Nov 03 03:12:03.830860 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.830792 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca" pod "168d9445-1e4f-4fd6-a05c-0c7ad890d519" (UID: "168d9445-1e4f-4fd6-a05c-0c7ad890d519"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:03.833744 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.833695 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca" pod "168d9445-1e4f-4fd6-a05c-0c7ad890d519" (UID: "168d9445-1e4f-4fd6-a05c-0c7ad890d519"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:03.850233 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.850164 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc" pod "168d9445-1e4f-4fd6-a05c-0c7ad890d519" (UID: "168d9445-1e4f-4fd6-a05c-0c7ad890d519"). InnerVolumeSpecName "kube-api-access-sq2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:03.926586 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.923196 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2sc\" (UniqueName: \"kubernetes.io/projected/168d9445-1e4f-4fd6-a05c-0c7ad890d519-kube-api-access-sq2sc\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:03.926586 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.923249 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:03.926586 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:03.923267 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168d9445-1e4f-4fd6-a05c-0c7ad890d519-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:04.486622 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:04.483590 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" event={"ID":"168d9445-1e4f-4fd6-a05c-0c7ad890d519","Type":"ContainerDied","Data":"12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27"} Nov 03 03:12:04.486622 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:04.483667 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e2a663e3406d530e3312a8a10b7b32f3d72c5c4c704608a1c5774545fcff27" Nov 03 03:12:04.486622 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:04.483756 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91d8f222-bwg4k" Nov 03 03:12:32.152834 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:12:32.152792 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:12:45.659301 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:45.659189 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:12:45.744638 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:45.744608 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6jftg_35bfbf47-81e9-4f0c-818c-965de586692b/console-operator/2.log" Nov 03 03:12:49.937682 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.937622 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs"] Nov 03 03:12:49.938438 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.938410 4992 cpu_manager.go:414] "containerMap: removing container" podUID="168d9445-1e4f-4fd6-a05c-0c7ad890d519" containerName="revoke-keys" Nov 03 03:12:49.938486 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.938443 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168d9445-1e4f-4fd6-a05c-0c7ad890d519" containerName="revoke-keys" Nov 03 03:12:49.938666 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.938651 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="168d9445-1e4f-4fd6-a05c-0c7ad890d519" containerName="revoke-keys" Nov 03 03:12:49.942672 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.942642 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:49.999792 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:49.999745 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs"] Nov 03 03:12:50.043527 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.043483 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:12:50.043751 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.043552 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:12:50.043751 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.043630 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:12:50.048394 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.048366 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:12:50.082036 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.081994 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.082036 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.082038 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29w7p\" (UniqueName: \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.082340 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.082279 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.183441 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.183397 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.183678 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.183456 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29w7p\" (UniqueName: \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.183678 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.183649 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.184370 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.184344 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.185948 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.185919 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.246950 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.246873 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29w7p\" (UniqueName: \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\") pod \"maas-api-revoke-sub-7e786030-8pwvs\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.256683 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.256650 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:50.633383 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:12:50.633343 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4521c884_90ce_420b_ac3f_37dd3cdae06d.slice/crio-e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c WatchSource:0}: Error finding container e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c: Status 404 returned error can't find the container with id e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c Nov 03 03:12:50.635201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.635171 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs"] Nov 03 03:12:50.851015 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:50.850973 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" event={"ID":"4521c884-90ce-420b-ac3f-37dd3cdae06d","Type":"ContainerStarted","Data":"e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c"} Nov 03 03:12:51.858044 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:51.857954 4992 generic.go:356] "Generic (PLEG): container finished" podID="4521c884-90ce-420b-ac3f-37dd3cdae06d" containerID="dd4e640a944053c11fc96f0c32a5d77e3e84bfe386dff889a785e1ce1ab0aedc" exitCode=0 Nov 03 03:12:51.858044 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:51.858018 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" event={"ID":"4521c884-90ce-420b-ac3f-37dd3cdae06d","Type":"ContainerDied","Data":"dd4e640a944053c11fc96f0c32a5d77e3e84bfe386dff889a785e1ce1ab0aedc"} Nov 03 03:12:53.001410 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.001382 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:53.115200 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.115099 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\") pod \"4521c884-90ce-420b-ac3f-37dd3cdae06d\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " Nov 03 03:12:53.115388 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.115220 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\") pod \"4521c884-90ce-420b-ac3f-37dd3cdae06d\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " Nov 03 03:12:53.115388 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.115252 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\" (UniqueName: \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\") pod \"4521c884-90ce-420b-ac3f-37dd3cdae06d\" (UID: \"4521c884-90ce-420b-ac3f-37dd3cdae06d\") " Nov 03 03:12:53.115610 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.115553 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca" pod "4521c884-90ce-420b-ac3f-37dd3cdae06d" (UID: "4521c884-90ce-420b-ac3f-37dd3cdae06d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:12:53.118155 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.118123 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca" pod "4521c884-90ce-420b-ac3f-37dd3cdae06d" (UID: "4521c884-90ce-420b-ac3f-37dd3cdae06d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:12:53.118155 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.118140 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p" pod "4521c884-90ce-420b-ac3f-37dd3cdae06d" (UID: "4521c884-90ce-420b-ac3f-37dd3cdae06d"). InnerVolumeSpecName "kube-api-access-29w7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:12:53.216865 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.216826 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:53.216865 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.216858 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29w7p\" (UniqueName: \"kubernetes.io/projected/4521c884-90ce-420b-ac3f-37dd3cdae06d-kube-api-access-29w7p\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:53.216865 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.216870 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4521c884-90ce-420b-ac3f-37dd3cdae06d-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:12:53.869472 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.869439 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" Nov 03 03:12:53.869707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.869474 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e786030-8pwvs" event={"ID":"4521c884-90ce-420b-ac3f-37dd3cdae06d","Type":"ContainerDied","Data":"e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c"} Nov 03 03:12:53.869707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:12:53.869523 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14dab9f501d4d8b90116d2fa169823d316a646307028a6761e76131a75ab20c" Nov 03 03:13:00.948765 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.948728 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk"] Nov 03 03:13:00.949271 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.949251 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4521c884-90ce-420b-ac3f-37dd3cdae06d" containerName="revoke-keys" Nov 03 03:13:00.949338 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.949272 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4521c884-90ce-420b-ac3f-37dd3cdae06d" containerName="revoke-keys" Nov 03 03:13:00.949447 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.949424 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4521c884-90ce-420b-ac3f-37dd3cdae06d" containerName="revoke-keys" Nov 03 03:13:00.952818 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.952797 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:00.992800 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.992761 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:00.992976 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.992902 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:00.992976 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:00.992937 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mdf\" (UniqueName: \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.007081 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.007046 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk"] Nov 03 03:13:01.056548 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.056497 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:01.056548 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.056498 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:01.093829 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.093788 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.094028 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.093846 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mdf\" (UniqueName: \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.094028 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.093891 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.096260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.096227 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.108469 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.108442 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:01.108637 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.108463 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:01.115099 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.115072 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.157751 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.157715 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mdf\" (UniqueName: \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\") pod \"maas-api-revoke-sub-961a4af6-r4bnk\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.270805 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.270709 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:01.649620 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:01.649578 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd365c88d_d2a7_4bd0_95ef_74f146bbb4b4.slice/crio-aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162 WatchSource:0}: Error finding container aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162: Status 404 returned error can't find the container with id aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162 Nov 03 03:13:01.651950 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.651927 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk"] Nov 03 03:13:01.911079 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:01.910992 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" event={"ID":"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4","Type":"ContainerStarted","Data":"aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162"} Nov 03 03:13:02.151358 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:13:02.151324 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:13:02.922476 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:02.922433 4992 generic.go:356] "Generic (PLEG): container finished" podID="d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" containerID="62b41aa63a8cf19a4df11311d2a0bbd16ad3867b1827474e5f283f753ae49cb3" exitCode=0 Nov 03 03:13:02.922709 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:02.922522 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" event={"ID":"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4","Type":"ContainerDied","Data":"62b41aa63a8cf19a4df11311d2a0bbd16ad3867b1827474e5f283f753ae49cb3"} Nov 03 03:13:04.071235 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.071204 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:04.116943 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.116898 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\") pod \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " Nov 03 03:13:04.117167 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.117019 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\" (UniqueName: \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\") pod \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " Nov 03 03:13:04.117167 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.117113 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\") pod \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\" (UID: \"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4\") " Nov 03 03:13:04.117448 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.117421 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca" pod "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" (UID: "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:04.119187 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.119163 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca" pod "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" (UID: "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:04.119315 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.119226 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf" pod "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" (UID: "d365c88d-d2a7-4bd0-95ef-74f146bbb4b4"). InnerVolumeSpecName "kube-api-access-q9mdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:04.218027 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.217931 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:04.218027 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.217969 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mdf\" (UniqueName: \"kubernetes.io/projected/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-kube-api-access-q9mdf\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:04.218027 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.217981 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365c88d-d2a7-4bd0-95ef-74f146bbb4b4-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:04.936934 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.936890 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" event={"ID":"d365c88d-d2a7-4bd0-95ef-74f146bbb4b4","Type":"ContainerDied","Data":"aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162"} Nov 03 03:13:04.937242 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.937212 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab521ee08e8858346af51d220077c1704489498a16db6f6730ac235b683d162" Nov 03 03:13:04.937373 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:04.937291 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961a4af6-r4bnk" Nov 03 03:13:11.812170 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.812132 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g"] Nov 03 03:13:11.812734 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.812607 4992 cpu_manager.go:414] "containerMap: removing container" podUID="d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" containerName="revoke-keys" Nov 03 03:13:11.812734 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.812621 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" containerName="revoke-keys" Nov 03 03:13:11.812734 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.812704 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="d365c88d-d2a7-4bd0-95ef-74f146bbb4b4" containerName="revoke-keys" Nov 03 03:13:11.815951 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.815931 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.870193 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.870155 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g"] Nov 03 03:13:11.895388 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.895345 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bnf\" (UniqueName: \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.895598 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.895397 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.895598 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.895500 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.919591 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.919526 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:11.919797 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.919680 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:11.956598 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.956550 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:11.956768 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.956602 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:11.996181 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.996144 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bnf\" (UniqueName: \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.996364 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.996192 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.996364 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.996254 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.996963 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.996940 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:11.998703 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:11.998681 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:12.059501 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:12.059469 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bnf\" (UniqueName: \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\") pod \"maas-api-revoke-sub-41cbdc9b-htm6g\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:12.131009 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:12.130923 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:12.503509 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:12.503474 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa15afd_6dd1_47c2_8297_1b75085f1410.slice/crio-8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4 WatchSource:0}: Error finding container 8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4: Status 404 returned error can't find the container with id 8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4 Nov 03 03:13:12.506112 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:12.506080 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g"] Nov 03 03:13:12.982887 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:12.982842 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" event={"ID":"8aa15afd-6dd1-47c2-8297-1b75085f1410","Type":"ContainerStarted","Data":"b1bcac287996b5397447561e2fcb18b3bea94d7c32ce0ec156788b8f707e2391"} Nov 03 03:13:12.983305 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:12.982914 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" event={"ID":"8aa15afd-6dd1-47c2-8297-1b75085f1410","Type":"ContainerStarted","Data":"8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4"} Nov 03 03:13:13.103545 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:13.103437 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" podStartSLOduration=2.10341962 podStartE2EDuration="2.10341962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:13.102452399 +0000 UTC m=+6928.681275000" watchObservedRunningTime="2026-11-03 03:13:13.10341962 +0000 UTC m=+6928.682242201" Nov 03 03:13:13.989240 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:13.989203 4992 generic.go:356] "Generic (PLEG): container finished" podID="8aa15afd-6dd1-47c2-8297-1b75085f1410" containerID="b1bcac287996b5397447561e2fcb18b3bea94d7c32ce0ec156788b8f707e2391" exitCode=0 Nov 03 03:13:13.989691 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:13.989286 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" event={"ID":"8aa15afd-6dd1-47c2-8297-1b75085f1410","Type":"ContainerDied","Data":"b1bcac287996b5397447561e2fcb18b3bea94d7c32ce0ec156788b8f707e2391"} Nov 03 03:13:15.142808 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.142778 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:15.228179 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.228143 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\") pod \"8aa15afd-6dd1-47c2-8297-1b75085f1410\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " Nov 03 03:13:15.228399 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.228247 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\" (UniqueName: \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\") pod \"8aa15afd-6dd1-47c2-8297-1b75085f1410\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " Nov 03 03:13:15.228399 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.228290 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\") pod \"8aa15afd-6dd1-47c2-8297-1b75085f1410\" (UID: \"8aa15afd-6dd1-47c2-8297-1b75085f1410\") " Nov 03 03:13:15.228619 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.228592 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca" pod "8aa15afd-6dd1-47c2-8297-1b75085f1410" (UID: "8aa15afd-6dd1-47c2-8297-1b75085f1410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:15.230345 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.230306 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf" pod "8aa15afd-6dd1-47c2-8297-1b75085f1410" (UID: "8aa15afd-6dd1-47c2-8297-1b75085f1410"). InnerVolumeSpecName "kube-api-access-j9bnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:15.230464 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.230435 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca" pod "8aa15afd-6dd1-47c2-8297-1b75085f1410" (UID: "8aa15afd-6dd1-47c2-8297-1b75085f1410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:15.329703 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.329606 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:15.329703 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.329641 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bnf\" (UniqueName: \"kubernetes.io/projected/8aa15afd-6dd1-47c2-8297-1b75085f1410-kube-api-access-j9bnf\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:15.329703 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:15.329651 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa15afd-6dd1-47c2-8297-1b75085f1410-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:16.000644 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:16.000598 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" event={"ID":"8aa15afd-6dd1-47c2-8297-1b75085f1410","Type":"ContainerDied","Data":"8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4"} Nov 03 03:13:16.000857 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:16.000659 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8356e11d764b2f464367d08d6c08e3a512f871e778890126d313acc69ff7b6f4" Nov 03 03:13:16.000857 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:16.000616 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cbdc9b-htm6g" Nov 03 03:13:22.778831 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.778794 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw"] Nov 03 03:13:22.779257 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.779207 4992 cpu_manager.go:414] "containerMap: removing container" podUID="8aa15afd-6dd1-47c2-8297-1b75085f1410" containerName="revoke-keys" Nov 03 03:13:22.779257 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.779219 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa15afd-6dd1-47c2-8297-1b75085f1410" containerName="revoke-keys" Nov 03 03:13:22.779350 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.779324 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa15afd-6dd1-47c2-8297-1b75085f1410" containerName="revoke-keys" Nov 03 03:13:22.782768 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.782745 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:22.836965 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.836929 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw"] Nov 03 03:13:22.837178 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.837028 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:22.885797 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.885751 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:22.885797 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.885758 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:22.888526 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.888504 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:22.907796 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.907756 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:22.907949 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.907869 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6jj\" (UniqueName: \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:22.907949 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:22.907917 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.009446 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.009404 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.009698 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.009464 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6jj\" (UniqueName: \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.009698 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.009493 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.010158 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.010135 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.012018 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.011993 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.072912 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.072840 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6jj\" (UniqueName: \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\") pod \"maas-api-revoke-sub-15816fcf-tlbmw\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.096331 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.096300 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:23.467473 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:23.467429 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4355b819_0af1_4c14_ac17_972eb50913c7.slice/crio-c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404 WatchSource:0}: Error finding container c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404: Status 404 returned error can't find the container with id c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404 Nov 03 03:13:23.470766 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:23.470742 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw"] Nov 03 03:13:24.044798 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:24.044711 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" event={"ID":"4355b819-0af1-4c14-ac17-972eb50913c7","Type":"ContainerStarted","Data":"3ac42e1a322fc3537cc479f41dbe026954df61b4c4d4d8b46c2948919d8bf4c1"} Nov 03 03:13:24.044798 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:24.044768 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" event={"ID":"4355b819-0af1-4c14-ac17-972eb50913c7","Type":"ContainerStarted","Data":"c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404"} Nov 03 03:13:24.166236 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:24.166172 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" podStartSLOduration=2.166155508 podStartE2EDuration="2.166155508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:24.164273457 +0000 UTC m=+6939.743096043" watchObservedRunningTime="2026-11-03 03:13:24.166155508 +0000 UTC m=+6939.744978106" Nov 03 03:13:25.051364 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:25.051325 4992 generic.go:356] "Generic (PLEG): container finished" podID="4355b819-0af1-4c14-ac17-972eb50913c7" containerID="3ac42e1a322fc3537cc479f41dbe026954df61b4c4d4d8b46c2948919d8bf4c1" exitCode=0 Nov 03 03:13:25.051812 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:25.051411 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" event={"ID":"4355b819-0af1-4c14-ac17-972eb50913c7","Type":"ContainerDied","Data":"3ac42e1a322fc3537cc479f41dbe026954df61b4c4d4d8b46c2948919d8bf4c1"} Nov 03 03:13:26.205763 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.205737 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:26.342336 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.342251 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\" (UniqueName: \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\") pod \"4355b819-0af1-4c14-ac17-972eb50913c7\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " Nov 03 03:13:26.342518 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.342427 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\") pod \"4355b819-0af1-4c14-ac17-972eb50913c7\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " Nov 03 03:13:26.342518 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.342452 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\") pod \"4355b819-0af1-4c14-ac17-972eb50913c7\" (UID: \"4355b819-0af1-4c14-ac17-972eb50913c7\") " Nov 03 03:13:26.342898 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.342868 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca" pod "4355b819-0af1-4c14-ac17-972eb50913c7" (UID: "4355b819-0af1-4c14-ac17-972eb50913c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:26.344603 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.344576 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca" pod "4355b819-0af1-4c14-ac17-972eb50913c7" (UID: "4355b819-0af1-4c14-ac17-972eb50913c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:26.344603 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.344589 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj" pod "4355b819-0af1-4c14-ac17-972eb50913c7" (UID: "4355b819-0af1-4c14-ac17-972eb50913c7"). InnerVolumeSpecName "kube-api-access-dh6jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:26.443361 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.443320 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:26.443361 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.443353 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4355b819-0af1-4c14-ac17-972eb50913c7-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:26.443361 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:26.443365 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6jj\" (UniqueName: \"kubernetes.io/projected/4355b819-0af1-4c14-ac17-972eb50913c7-kube-api-access-dh6jj\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:27.064477 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:27.064442 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" Nov 03 03:13:27.064723 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:27.064483 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15816fcf-tlbmw" event={"ID":"4355b819-0af1-4c14-ac17-972eb50913c7","Type":"ContainerDied","Data":"c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404"} Nov 03 03:13:27.064723 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:27.064615 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c716cf896c0f9df34a0f280c416a909aa833d4d2a4e0b1b2d3cd679db9145404" Nov 03 03:13:29.168723 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:29.168675 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nd2gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:13:29.169378 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:29.169336 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nd2gf" podUID="33773d82-e52c-4603-899c-6935a2db93df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:13:32.150415 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:13:32.150369 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:13:33.510653 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.510594 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt"] Nov 03 03:13:33.511448 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.511424 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4355b819-0af1-4c14-ac17-972eb50913c7" containerName="revoke-keys" Nov 03 03:13:33.511517 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.511454 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4355b819-0af1-4c14-ac17-972eb50913c7" containerName="revoke-keys" Nov 03 03:13:33.511654 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.511636 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4355b819-0af1-4c14-ac17-972eb50913c7" containerName="revoke-keys" Nov 03 03:13:33.515766 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.515740 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.566600 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.566551 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt"] Nov 03 03:13:33.615628 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.615592 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:33.615628 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.615630 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:33.621692 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.621648 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:33.621755 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.621705 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:33.637378 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.637347 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.637473 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.637403 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9q4c\" (UniqueName: \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.637553 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.637525 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.738634 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.738591 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.738826 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.738657 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9q4c\" (UniqueName: \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.738826 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.738715 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.739313 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.739288 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.741348 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.741321 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.804892 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.804811 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9q4c\" (UniqueName: \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\") pod \"maas-api-revoke-sub-673b9d5c-94jrt\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:33.831290 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:33.831249 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:34.207474 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:34.207434 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4a41d6_fe27_4afb_9cc5_9bb1e40542fb.slice/crio-13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6 WatchSource:0}: Error finding container 13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6: Status 404 returned error can't find the container with id 13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6 Nov 03 03:13:34.209141 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:34.209116 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt"] Nov 03 03:13:35.147318 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:35.147282 4992 generic.go:356] "Generic (PLEG): container finished" podID="bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" containerID="f97b4924d3747625a65019e326f1342fdbcd3b15d1010da0ec6350394c1cf47d" exitCode=0 Nov 03 03:13:35.147742 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:35.147347 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" event={"ID":"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb","Type":"ContainerDied","Data":"f97b4924d3747625a65019e326f1342fdbcd3b15d1010da0ec6350394c1cf47d"} Nov 03 03:13:35.147742 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:35.147384 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" event={"ID":"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb","Type":"ContainerStarted","Data":"13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6"} Nov 03 03:13:36.295499 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.295472 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:36.465311 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.465275 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\" (UniqueName: \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\") pod \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " Nov 03 03:13:36.465506 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.465327 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\") pod \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " Nov 03 03:13:36.465506 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.465347 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\") pod \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\" (UID: \"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb\") " Nov 03 03:13:36.465817 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.465773 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca" pod "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" (UID: "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:36.467570 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.467532 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca" pod "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" (UID: "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:36.467705 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.467629 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c" pod "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" (UID: "bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb"). InnerVolumeSpecName "kube-api-access-p9q4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:36.566825 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.566787 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9q4c\" (UniqueName: \"kubernetes.io/projected/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-kube-api-access-p9q4c\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:36.566825 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.566821 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:36.567039 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:36.566838 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:37.159674 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:37.159640 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" Nov 03 03:13:37.159857 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:37.159639 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-673b9d5c-94jrt" event={"ID":"bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb","Type":"ContainerDied","Data":"13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6"} Nov 03 03:13:37.159857 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:37.159776 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a532a1c9394145f9b5c20a0cb4ac7f8c6fe4e7afd1e105154707cf0f7526d6" Nov 03 03:13:44.354697 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.354654 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2"] Nov 03 03:13:44.355201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.355104 4992 cpu_manager.go:414] "containerMap: removing container" podUID="bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" containerName="revoke-keys" Nov 03 03:13:44.355201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.355116 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" containerName="revoke-keys" Nov 03 03:13:44.355201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.355194 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb" containerName="revoke-keys" Nov 03 03:13:44.358371 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.358352 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.415109 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.415069 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2"] Nov 03 03:13:44.447433 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.447396 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.447648 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.447462 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtx2\" (UniqueName: \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.447648 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.447599 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.463585 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.463523 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:44.463585 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.463523 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:44.463585 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.463525 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:44.463585 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.463586 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:44.548288 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.548250 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.548459 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.548328 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtx2\" (UniqueName: \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.548459 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.548373 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.549070 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.549050 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.550741 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.550719 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:44.671272 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.671240 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:44.681177 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:44.681147 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtx2\" (UniqueName: \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\") pod \"maas-api-revoke-sub-6776df4f-hrll2\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:45.024981 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:45.024894 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:45.031976 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:45.031951 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:45.407618 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:45.407585 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a948d7_e1bb_4f77_bb15_0dfe5cebf013.slice/crio-7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c WatchSource:0}: Error finding container 7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c: Status 404 returned error can't find the container with id 7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c Nov 03 03:13:45.410055 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:45.410027 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2"] Nov 03 03:13:46.209941 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:46.209899 4992 generic.go:356] "Generic (PLEG): container finished" podID="50a948d7-e1bb-4f77-bb15-0dfe5cebf013" containerID="4fe2a4b0d90a4cc12b1d66ecf6d39e97444c4d325b25bdf2bfe615b04988ac3a" exitCode=0 Nov 03 03:13:46.210117 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:46.209980 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" event={"ID":"50a948d7-e1bb-4f77-bb15-0dfe5cebf013","Type":"ContainerDied","Data":"4fe2a4b0d90a4cc12b1d66ecf6d39e97444c4d325b25bdf2bfe615b04988ac3a"} Nov 03 03:13:46.210117 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:46.210078 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" event={"ID":"50a948d7-e1bb-4f77-bb15-0dfe5cebf013","Type":"ContainerStarted","Data":"7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c"} Nov 03 03:13:47.420478 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.420440 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:47.479462 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.479360 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\") pod \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " Nov 03 03:13:47.479675 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.479483 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\") pod \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " Nov 03 03:13:47.479675 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.479515 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\" (UniqueName: \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\") pod \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\" (UID: \"50a948d7-e1bb-4f77-bb15-0dfe5cebf013\") " Nov 03 03:13:47.479988 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.479953 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca" pod "50a948d7-e1bb-4f77-bb15-0dfe5cebf013" (UID: "50a948d7-e1bb-4f77-bb15-0dfe5cebf013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:47.482620 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.482587 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca" pod "50a948d7-e1bb-4f77-bb15-0dfe5cebf013" (UID: "50a948d7-e1bb-4f77-bb15-0dfe5cebf013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:47.482733 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.482705 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2" pod "50a948d7-e1bb-4f77-bb15-0dfe5cebf013" (UID: "50a948d7-e1bb-4f77-bb15-0dfe5cebf013"). InnerVolumeSpecName "kube-api-access-qbtx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:47.581204 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.581154 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:47.581204 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.581200 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtx2\" (UniqueName: \"kubernetes.io/projected/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-kube-api-access-qbtx2\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:47.581492 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:47.581217 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50a948d7-e1bb-4f77-bb15-0dfe5cebf013-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:48.234860 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:48.234830 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" Nov 03 03:13:48.235810 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:48.234893 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6776df4f-hrll2" event={"ID":"50a948d7-e1bb-4f77-bb15-0dfe5cebf013","Type":"ContainerDied","Data":"7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c"} Nov 03 03:13:48.236011 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:48.235993 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab8a3bea9351e7bbf08ee10d90290af89a26596f34834a5986609698a18656c" Nov 03 03:13:55.160108 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.160032 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj"] Nov 03 03:13:55.160478 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.160464 4992 cpu_manager.go:414] "containerMap: removing container" podUID="50a948d7-e1bb-4f77-bb15-0dfe5cebf013" containerName="revoke-keys" Nov 03 03:13:55.160478 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.160475 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a948d7-e1bb-4f77-bb15-0dfe5cebf013" containerName="revoke-keys" Nov 03 03:13:55.160632 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.160620 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a948d7-e1bb-4f77-bb15-0dfe5cebf013" containerName="revoke-keys" Nov 03 03:13:55.163853 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.163834 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.202475 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.202421 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282wm\" (UniqueName: \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.202664 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.202500 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.202664 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.202634 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.218662 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.218628 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj"] Nov 03 03:13:55.266308 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.266274 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:13:55.266308 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.266275 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:13:55.266308 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.266292 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:13:55.269892 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.269864 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:13:55.303691 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.303658 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282wm\" (UniqueName: \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.303881 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.303704 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.303881 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.303732 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.304436 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.304405 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.306103 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.306080 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.367443 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.367404 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282wm\" (UniqueName: \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\") pod \"maas-api-revoke-sub-56d05f4f-2wbjj\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.482326 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.482281 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:55.854854 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:13:55.854815 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4f4bb9_f716_41b7_a4c5_aae7bb9378c1.slice/crio-c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4 WatchSource:0}: Error finding container c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4: Status 404 returned error can't find the container with id c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4 Nov 03 03:13:55.857487 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:55.857408 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj"] Nov 03 03:13:56.280207 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:56.280170 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" event={"ID":"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1","Type":"ContainerStarted","Data":"fbd5e44f9dc1cac8ea1990e7b69624b7e87d5f9337a407cb6089fdbc46ecee35"} Nov 03 03:13:56.280631 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:56.280230 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" event={"ID":"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1","Type":"ContainerStarted","Data":"c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4"} Nov 03 03:13:56.399684 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:56.399623 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" podStartSLOduration=1.399609418 podStartE2EDuration="1.399609418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:13:56.399276844 +0000 UTC m=+6971.978099430" watchObservedRunningTime="2026-11-03 03:13:56.399609418 +0000 UTC m=+6971.978432002" Nov 03 03:13:57.287886 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:57.287849 4992 generic.go:356] "Generic (PLEG): container finished" podID="4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" containerID="fbd5e44f9dc1cac8ea1990e7b69624b7e87d5f9337a407cb6089fdbc46ecee35" exitCode=0 Nov 03 03:13:57.288327 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:57.287930 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" event={"ID":"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1","Type":"ContainerDied","Data":"fbd5e44f9dc1cac8ea1990e7b69624b7e87d5f9337a407cb6089fdbc46ecee35"} Nov 03 03:13:58.483615 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.483588 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:13:58.538371 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.538321 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\") pod \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " Nov 03 03:13:58.538601 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.538452 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\" (UniqueName: \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\") pod \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " Nov 03 03:13:58.538601 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.538514 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\") pod \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\" (UID: \"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1\") " Nov 03 03:13:58.538806 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.538778 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca" pod "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" (UID: "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:13:58.538920 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.538890 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:58.541300 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.541271 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca" pod "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" (UID: "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:13:58.550217 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.550120 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm" pod "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" (UID: "4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1"). InnerVolumeSpecName "kube-api-access-282wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:13:58.644547 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.644484 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282wm\" (UniqueName: \"kubernetes.io/projected/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-kube-api-access-282wm\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:58.644547 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:58.644520 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:13:59.311765 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:59.311727 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" event={"ID":"4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1","Type":"ContainerDied","Data":"c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4"} Nov 03 03:13:59.317690 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:59.317648 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7b4b9dca83d07494ebdf50619c500394d3ac479f08fef1d1f5f84fc92c7c6f4" Nov 03 03:13:59.318141 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:13:59.312664 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d05f4f-2wbjj" Nov 03 03:14:02.150940 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:14:02.150902 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:14:05.954121 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:05.954080 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t"] Nov 03 03:14:05.954694 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:05.954571 4992 cpu_manager.go:414] "containerMap: removing container" podUID="4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" containerName="revoke-keys" Nov 03 03:14:05.954694 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:05.954598 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" containerName="revoke-keys" Nov 03 03:14:05.954694 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:05.954672 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1" containerName="revoke-keys" Nov 03 03:14:05.958112 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:05.958086 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.012391 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.012361 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t"] Nov 03 03:14:06.014039 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.014004 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.014178 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.014069 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.014277 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.014255 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpt9\" (UniqueName: \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.061249 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.061214 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:06.061249 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.061227 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:06.061470 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.061283 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:06.064241 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.064221 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:14:06.115144 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.115097 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpt9\" (UniqueName: \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.115357 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.115179 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.115357 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.115222 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.115975 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.115949 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.117706 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.117675 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.178704 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.178669 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpt9\" (UniqueName: \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\") pod \"maas-api-revoke-sub-b28b10d8-28m2t\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.272038 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.271955 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:06.644845 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:14:06.644805 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c6b447_2479_4e83_a09a_4c5d8c114de3.slice/crio-7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7 WatchSource:0}: Error finding container 7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7: Status 404 returned error can't find the container with id 7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7 Nov 03 03:14:06.647164 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:06.647140 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t"] Nov 03 03:14:07.365024 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:07.364927 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" event={"ID":"99c6b447-2479-4e83-a09a-4c5d8c114de3","Type":"ContainerStarted","Data":"1deb58ec9021a5b9071f40e926d62171bcb90cc35e89482b11d5453cbe53af06"} Nov 03 03:14:07.365446 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:07.365026 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" event={"ID":"99c6b447-2479-4e83-a09a-4c5d8c114de3","Type":"ContainerStarted","Data":"7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7"} Nov 03 03:14:07.485099 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:07.485041 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" podStartSLOduration=2.485026723 podStartE2EDuration="2.485026723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:07.48425818 +0000 UTC m=+6983.063080767" watchObservedRunningTime="2026-11-03 03:14:07.485026723 +0000 UTC m=+6983.063849301" Nov 03 03:14:08.373754 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:08.373714 4992 generic.go:356] "Generic (PLEG): container finished" podID="99c6b447-2479-4e83-a09a-4c5d8c114de3" containerID="1deb58ec9021a5b9071f40e926d62171bcb90cc35e89482b11d5453cbe53af06" exitCode=0 Nov 03 03:14:08.374295 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:08.373771 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" event={"ID":"99c6b447-2479-4e83-a09a-4c5d8c114de3","Type":"ContainerDied","Data":"1deb58ec9021a5b9071f40e926d62171bcb90cc35e89482b11d5453cbe53af06"} Nov 03 03:14:09.753587 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.753543 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:09.873981 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.873939 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\") pod \"99c6b447-2479-4e83-a09a-4c5d8c114de3\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " Nov 03 03:14:09.874285 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.874262 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\" (UniqueName: \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\") pod \"99c6b447-2479-4e83-a09a-4c5d8c114de3\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " Nov 03 03:14:09.874458 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.874439 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\") pod \"99c6b447-2479-4e83-a09a-4c5d8c114de3\" (UID: \"99c6b447-2479-4e83-a09a-4c5d8c114de3\") " Nov 03 03:14:09.877626 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.877318 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca" pod "99c6b447-2479-4e83-a09a-4c5d8c114de3" (UID: "99c6b447-2479-4e83-a09a-4c5d8c114de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:09.904580 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.904040 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9" pod "99c6b447-2479-4e83-a09a-4c5d8c114de3" (UID: "99c6b447-2479-4e83-a09a-4c5d8c114de3"). InnerVolumeSpecName "kube-api-access-xjpt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:09.915116 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.915053 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca" pod "99c6b447-2479-4e83-a09a-4c5d8c114de3" (UID: "99c6b447-2479-4e83-a09a-4c5d8c114de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:09.979366 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.979207 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:09.979366 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.979263 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpt9\" (UniqueName: \"kubernetes.io/projected/99c6b447-2479-4e83-a09a-4c5d8c114de3-kube-api-access-xjpt9\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:09.979366 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:09.979281 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99c6b447-2479-4e83-a09a-4c5d8c114de3-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:10.440626 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:10.440581 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" event={"ID":"99c6b447-2479-4e83-a09a-4c5d8c114de3","Type":"ContainerDied","Data":"7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7"} Nov 03 03:14:10.440817 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:10.440651 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fc53f349ebebb081ea8272afcd2f561d118268267f3ddd0bca3382d745c97c7" Nov 03 03:14:10.440817 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:10.440661 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28b10d8-28m2t" Nov 03 03:14:16.753881 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.753844 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2"] Nov 03 03:14:16.754378 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.754317 4992 cpu_manager.go:414] "containerMap: removing container" podUID="99c6b447-2479-4e83-a09a-4c5d8c114de3" containerName="revoke-keys" Nov 03 03:14:16.754378 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.754331 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c6b447-2479-4e83-a09a-4c5d8c114de3" containerName="revoke-keys" Nov 03 03:14:16.754498 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.754399 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c6b447-2479-4e83-a09a-4c5d8c114de3" containerName="revoke-keys" Nov 03 03:14:16.757840 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.757821 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.811584 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.811537 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2"] Nov 03 03:14:16.860443 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.860404 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:16.860657 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.860486 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:14:16.863782 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.863763 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:16.863842 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.863784 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:16.882899 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.882868 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.883048 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.882928 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvdd\" (UniqueName: \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.883048 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.883037 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.983849 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.983808 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvdd\" (UniqueName: \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.984037 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.983879 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.984037 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.983932 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.984505 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.984478 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:16.986283 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:16.986263 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:17.047257 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:17.047183 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvdd\" (UniqueName: \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\") pod \"maas-api-revoke-sub-132d83c3-mgvq2\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:17.072033 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:17.071996 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:17.443659 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:14:17.443622 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd23c856_24be_48c5_b79b_a704f7b6a8a8.slice/crio-10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648 WatchSource:0}: Error finding container 10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648: Status 404 returned error can't find the container with id 10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648 Nov 03 03:14:17.447606 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:17.447577 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2"] Nov 03 03:14:17.479835 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:17.479805 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" event={"ID":"fd23c856-24be-48c5-b79b-a704f7b6a8a8","Type":"ContainerStarted","Data":"10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648"} Nov 03 03:14:18.486941 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:18.486902 4992 generic.go:356] "Generic (PLEG): container finished" podID="fd23c856-24be-48c5-b79b-a704f7b6a8a8" containerID="22d9cd1347cd1e2dc0dd84cdea602b20b0f23a0483dd07d3a71175db159a31b0" exitCode=0 Nov 03 03:14:18.487341 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:18.486974 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" event={"ID":"fd23c856-24be-48c5-b79b-a704f7b6a8a8","Type":"ContainerDied","Data":"22d9cd1347cd1e2dc0dd84cdea602b20b0f23a0483dd07d3a71175db159a31b0"} Nov 03 03:14:19.656799 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.656749 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:19.723685 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.723640 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\" (UniqueName: \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\") pod \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " Nov 03 03:14:19.723887 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.723745 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\") pod \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " Nov 03 03:14:19.723887 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.723808 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\") pod \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\" (UID: \"fd23c856-24be-48c5-b79b-a704f7b6a8a8\") " Nov 03 03:14:19.724215 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.724186 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca" pod "fd23c856-24be-48c5-b79b-a704f7b6a8a8" (UID: "fd23c856-24be-48c5-b79b-a704f7b6a8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:19.725974 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.725948 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca" pod "fd23c856-24be-48c5-b79b-a704f7b6a8a8" (UID: "fd23c856-24be-48c5-b79b-a704f7b6a8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:19.726440 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.726423 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd" pod "fd23c856-24be-48c5-b79b-a704f7b6a8a8" (UID: "fd23c856-24be-48c5-b79b-a704f7b6a8a8"). InnerVolumeSpecName "kube-api-access-6kvdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:19.825396 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.825313 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:19.825396 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.825347 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd23c856-24be-48c5-b79b-a704f7b6a8a8-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:19.825396 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:19.825362 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvdd\" (UniqueName: \"kubernetes.io/projected/fd23c856-24be-48c5-b79b-a704f7b6a8a8-kube-api-access-6kvdd\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:20.533064 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:20.533026 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" event={"ID":"fd23c856-24be-48c5-b79b-a704f7b6a8a8","Type":"ContainerDied","Data":"10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648"} Nov 03 03:14:20.533435 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:20.533412 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10612c77d3805c8e1ebbf23f7d95b028e69fc48daca12f8bc67812752e9a3648" Nov 03 03:14:20.533770 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:20.533750 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-132d83c3-mgvq2" Nov 03 03:14:27.051396 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.051356 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2"] Nov 03 03:14:27.102928 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.102889 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84687dc3-d6ql2"] Nov 03 03:14:27.615389 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.615353 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk"] Nov 03 03:14:27.615830 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.615815 4992 cpu_manager.go:414] "containerMap: removing container" podUID="fd23c856-24be-48c5-b79b-a704f7b6a8a8" containerName="revoke-keys" Nov 03 03:14:27.615890 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.615832 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd23c856-24be-48c5-b79b-a704f7b6a8a8" containerName="revoke-keys" Nov 03 03:14:27.615939 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.615910 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd23c856-24be-48c5-b79b-a704f7b6a8a8" containerName="revoke-keys" Nov 03 03:14:27.619014 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.618995 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.673499 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.673468 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:27.674591 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.674539 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk"] Nov 03 03:14:27.710601 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.710537 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.710774 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.710697 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln74c\" (UniqueName: \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.710827 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.710808 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.722780 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.722733 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:27.722953 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.722842 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:14:27.722953 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.722870 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:27.811397 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.811360 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.811633 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.811419 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln74c\" (UniqueName: \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.811633 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.811475 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.812176 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.812153 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.813807 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.813785 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.874903 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.874828 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln74c\" (UniqueName: \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\") pod \"maas-api-revoke-sub-b83bf462-925zk\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:27.932471 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:27.932440 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:28.303572 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:14:28.303523 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3b4855_f92e_4431_b173_2fd7da55f166.slice/crio-c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0 WatchSource:0}: Error finding container c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0: Status 404 returned error can't find the container with id c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0 Nov 03 03:14:28.305941 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:28.305916 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk"] Nov 03 03:14:28.580834 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:28.580797 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" event={"ID":"0e3b4855-f92e-4431-b173-2fd7da55f166","Type":"ContainerStarted","Data":"c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0"} Nov 03 03:14:28.593418 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:28.593389 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f02395-bb1e-4f09-bc3a-2669c96805f2" path="/var/lib/kubelet/pods/d2f02395-bb1e-4f09-bc3a-2669c96805f2/volumes" Nov 03 03:14:29.587425 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:29.587382 4992 generic.go:356] "Generic (PLEG): container finished" podID="0e3b4855-f92e-4431-b173-2fd7da55f166" containerID="507a7d3c76d9be61358a0ebcffd1019c2e9594fabd09562be04201aac0a3092c" exitCode=0 Nov 03 03:14:29.587906 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:29.587467 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" event={"ID":"0e3b4855-f92e-4431-b173-2fd7da55f166","Type":"ContainerDied","Data":"507a7d3c76d9be61358a0ebcffd1019c2e9594fabd09562be04201aac0a3092c"} Nov 03 03:14:30.758289 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.758245 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:30.844087 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.844035 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\" (UniqueName: \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\") pod \"0e3b4855-f92e-4431-b173-2fd7da55f166\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " Nov 03 03:14:30.844280 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.844105 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\") pod \"0e3b4855-f92e-4431-b173-2fd7da55f166\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " Nov 03 03:14:30.844280 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.844241 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\") pod \"0e3b4855-f92e-4431-b173-2fd7da55f166\" (UID: \"0e3b4855-f92e-4431-b173-2fd7da55f166\") " Nov 03 03:14:30.844644 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.844604 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca" pod "0e3b4855-f92e-4431-b173-2fd7da55f166" (UID: "0e3b4855-f92e-4431-b173-2fd7da55f166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:30.846613 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.846515 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca" pod "0e3b4855-f92e-4431-b173-2fd7da55f166" (UID: "0e3b4855-f92e-4431-b173-2fd7da55f166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:30.846911 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.846876 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c" pod "0e3b4855-f92e-4431-b173-2fd7da55f166" (UID: "0e3b4855-f92e-4431-b173-2fd7da55f166"). InnerVolumeSpecName "kube-api-access-ln74c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:30.945946 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.945846 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:30.945946 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.945895 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3b4855-f92e-4431-b173-2fd7da55f166-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:30.945946 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:30.945916 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln74c\" (UniqueName: \"kubernetes.io/projected/0e3b4855-f92e-4431-b173-2fd7da55f166-kube-api-access-ln74c\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:31.632507 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:31.632464 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" event={"ID":"0e3b4855-f92e-4431-b173-2fd7da55f166","Type":"ContainerDied","Data":"c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0"} Nov 03 03:14:31.632854 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:31.632832 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4cd2f2342b2ec3e936a5fa07b2045306dd97f9b71b4346985fc5fb1740e2ad0" Nov 03 03:14:31.633060 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:31.633044 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b83bf462-925zk" Nov 03 03:14:32.159572 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:14:32.159523 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:14:38.060064 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.060023 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q"] Nov 03 03:14:38.111698 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.111659 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81de716-wsl5q"] Nov 03 03:14:38.400167 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.400082 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb"] Nov 03 03:14:38.400551 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.400535 4992 cpu_manager.go:414] "containerMap: removing container" podUID="0e3b4855-f92e-4431-b173-2fd7da55f166" containerName="revoke-keys" Nov 03 03:14:38.400611 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.400554 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3b4855-f92e-4431-b173-2fd7da55f166" containerName="revoke-keys" Nov 03 03:14:38.400658 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.400648 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3b4855-f92e-4431-b173-2fd7da55f166" containerName="revoke-keys" Nov 03 03:14:38.403935 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.403916 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.458947 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.458910 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:38.459939 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.459915 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb"] Nov 03 03:14:38.507921 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.507885 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:14:38.507921 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.507889 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:38.509914 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.509892 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:38.544188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.544151 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.544188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.544191 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9h5\" (UniqueName: \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.544417 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.544302 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.594276 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.594240 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ec39c9-500e-4036-bf0a-b9bf3abf143e" path="/var/lib/kubelet/pods/38ec39c9-500e-4036-bf0a-b9bf3abf143e/volumes" Nov 03 03:14:38.645678 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.645645 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.645678 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.645685 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9h5\" (UniqueName: \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.645915 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.645748 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.646354 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.646329 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.648171 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.648147 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.710227 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.710190 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9h5\" (UniqueName: \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\") pod \"maas-api-revoke-sub-050f0c64-t5nlb\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:38.717016 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:38.716993 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:39.094327 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:14:39.094286 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0798d5_8bda_43a8_b8d4_b09f6a52ef0c.slice/crio-28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc WatchSource:0}: Error finding container 28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc: Status 404 returned error can't find the container with id 28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc Nov 03 03:14:39.096755 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:39.096728 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb"] Nov 03 03:14:39.684193 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:39.684155 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" event={"ID":"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c","Type":"ContainerStarted","Data":"3ac2e1455caa530eacde11655f0517b137a920c00379f71216b02c3b8d39f424"} Nov 03 03:14:39.684389 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:39.684209 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" event={"ID":"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c","Type":"ContainerStarted","Data":"28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc"} Nov 03 03:14:39.805398 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:39.805339 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" podStartSLOduration=1.805322316 podStartE2EDuration="1.805322316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:39.803812129 +0000 UTC m=+7015.382634715" watchObservedRunningTime="2026-11-03 03:14:39.805322316 +0000 UTC m=+7015.384144935" Nov 03 03:14:40.691688 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:40.691654 4992 generic.go:356] "Generic (PLEG): container finished" podID="1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" containerID="3ac2e1455caa530eacde11655f0517b137a920c00379f71216b02c3b8d39f424" exitCode=0 Nov 03 03:14:40.692201 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:40.691724 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" event={"ID":"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c","Type":"ContainerDied","Data":"3ac2e1455caa530eacde11655f0517b137a920c00379f71216b02c3b8d39f424"} Nov 03 03:14:42.131587 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.130845 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:42.228596 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.216858 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\" (UniqueName: \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\") pod \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " Nov 03 03:14:42.228596 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.216939 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\") pod \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " Nov 03 03:14:42.228596 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.217120 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\") pod \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\" (UID: \"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c\") " Nov 03 03:14:42.228596 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.221779 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca" pod "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" (UID: "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:42.238188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.237873 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5" pod "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" (UID: "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c"). InnerVolumeSpecName "kube-api-access-vl9h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:42.241052 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.241016 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca" pod "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" (UID: "1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:42.320286 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.319248 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9h5\" (UniqueName: \"kubernetes.io/projected/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-kube-api-access-vl9h5\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:42.320286 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.319294 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:42.320286 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.319310 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:42.720925 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.720879 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" event={"ID":"1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c","Type":"ContainerDied","Data":"28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc"} Nov 03 03:14:42.721113 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.720952 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28f72c90b66b8b5650ac88f65d7023f37db2d3a06ce1943630dbdb73c73626fc" Nov 03 03:14:42.721113 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:42.721024 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-050f0c64-t5nlb" Nov 03 03:14:43.520448 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:43.520403 4992 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nf7vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 03:14:43.520990 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:43.520493 4992 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nf7vp" podUID="0277aa76-953c-4912-861b-c1e841b0a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 03:14:49.193790 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.193746 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj"] Nov 03 03:14:49.194316 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.194235 4992 cpu_manager.go:414] "containerMap: removing container" podUID="1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" containerName="revoke-keys" Nov 03 03:14:49.194316 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.194249 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" containerName="revoke-keys" Nov 03 03:14:49.194397 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.194328 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c" containerName="revoke-keys" Nov 03 03:14:49.198859 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.198833 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.251522 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.251480 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj"] Nov 03 03:14:49.294219 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.294180 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxtf\" (UniqueName: \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.294415 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.294291 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.294415 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.294351 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.300256 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.300222 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 03:14:49.300256 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.300236 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:14:49.300444 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.300242 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:14:49.300444 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.300314 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:14:49.395033 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.394993 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.395240 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.395094 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxtf\" (UniqueName: \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.395240 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.395137 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.395793 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.395766 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.397515 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.397491 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.458805 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.458720 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxtf\" (UniqueName: \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\") pod \"maas-api-revoke-sub-fb200804-mvpzj\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.512191 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.512145 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:49.888511 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:14:49.888472 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0ccdf2_4c4f_4711_bf50_9fce222c4c56.slice/crio-598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c WatchSource:0}: Error finding container 598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c: Status 404 returned error can't find the container with id 598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c Nov 03 03:14:49.890641 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:49.890612 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj"] Nov 03 03:14:50.766707 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:50.766665 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" event={"ID":"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56","Type":"ContainerStarted","Data":"c0f9058cb8a34c9c574693ab2479b47d38f0ee5f5a9f5098ccdd8c86a91618b8"} Nov 03 03:14:50.767151 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:50.766736 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" event={"ID":"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56","Type":"ContainerStarted","Data":"598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c"} Nov 03 03:14:50.886847 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:50.886790 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" podStartSLOduration=1.88677743 podStartE2EDuration="1.88677743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:14:50.88666551 +0000 UTC m=+7026.465488097" watchObservedRunningTime="2026-11-03 03:14:50.88677743 +0000 UTC m=+7026.465600017" Nov 03 03:14:51.820595 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:51.817552 4992 generic.go:356] "Generic (PLEG): container finished" podID="ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" containerID="c0f9058cb8a34c9c574693ab2479b47d38f0ee5f5a9f5098ccdd8c86a91618b8" exitCode=0 Nov 03 03:14:51.820595 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:51.817713 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" event={"ID":"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56","Type":"ContainerDied","Data":"c0f9058cb8a34c9c574693ab2479b47d38f0ee5f5a9f5098ccdd8c86a91618b8"} Nov 03 03:14:53.239613 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.239059 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:53.254087 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.254034 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\") pod \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " Nov 03 03:14:53.254087 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.254091 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\") pod \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " Nov 03 03:14:53.254313 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.254125 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\" (UniqueName: \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\") pod \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\" (UID: \"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56\") " Nov 03 03:14:53.254713 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.254676 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca" pod "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" (UID: "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:14:53.256662 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.256623 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca" pod "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" (UID: "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 03:14:53.256781 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.256660 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf" pod "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" (UID: "ef0ccdf2-4c4f-4711-bf50-9fce222c4c56"). InnerVolumeSpecName "kube-api-access-rcxtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:14:53.355710 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.355662 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-openshift-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:53.355710 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.355709 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-maas-api-xks-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:53.355710 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.355720 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcxtf\" (UniqueName: \"kubernetes.io/projected/ef0ccdf2-4c4f-4711-bf50-9fce222c4c56-kube-api-access-rcxtf\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:14:53.840053 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.840011 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" Nov 03 03:14:53.840053 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.840029 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb200804-mvpzj" event={"ID":"ef0ccdf2-4c4f-4711-bf50-9fce222c4c56","Type":"ContainerDied","Data":"598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c"} Nov 03 03:14:53.840290 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:14:53.840080 4992 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598360a129eba7260560f1877ce21d02363be248165130b55c7c8e8f0dd3744c" Nov 03 03:15:00.163256 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.163214 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:00.163898 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.163876 4992 cpu_manager.go:414] "containerMap: removing container" podUID="ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" containerName="revoke-keys" Nov 03 03:15:00.163994 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.163900 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" containerName="revoke-keys" Nov 03 03:15:00.164072 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.164058 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0ccdf2-4c4f-4711-bf50-9fce222c4c56" containerName="revoke-keys" Nov 03 03:15:00.167322 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.167295 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.219188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.219146 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbsz\" (UniqueName: \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.219382 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.219326 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.221098 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.221066 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:00.269518 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.269434 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 03:15:00.269781 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.269581 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 03:15:00.269781 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.269658 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\"" Nov 03 03:15:00.320404 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.320367 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbsz\" (UniqueName: \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.320610 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.320471 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.321357 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.321279 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.383617 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.383583 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbsz\" (UniqueName: \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\") pod \"maas-api-key-cleanup-29894595-q8grs\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.480727 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.480631 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:00.854061 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:15:00.854021 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b3ca6e_0f22_4292_837e_a3ad7858de2c.slice/crio-c77de6e0035626f38d88ae1de9ccf8fd36e4f31628d6d73e351f0a612d609af7 WatchSource:0}: Error finding container c77de6e0035626f38d88ae1de9ccf8fd36e4f31628d6d73e351f0a612d609af7: Status 404 returned error can't find the container with id c77de6e0035626f38d88ae1de9ccf8fd36e4f31628d6d73e351f0a612d609af7 Nov 03 03:15:00.856763 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.856663 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:00.882618 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:00.882584 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerStarted","Data":"c77de6e0035626f38d88ae1de9ccf8fd36e4f31628d6d73e351f0a612d609af7"} Nov 03 03:15:01.889655 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:01.889615 4992 generic.go:356] "Generic (PLEG): container finished" podID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerID="de32c20818cd80739bbd420d555bceb5689a545a09dead7aa42073784dca19bd" exitCode=60 Nov 03 03:15:01.890060 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:01.889682 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerDied","Data":"de32c20818cd80739bbd420d555bceb5689a545a09dead7aa42073784dca19bd"} Nov 03 03:15:01.890060 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:01.889972 4992 scope.go:122] "RemoveContainer" containerID="de32c20818cd80739bbd420d555bceb5689a545a09dead7aa42073784dca19bd" Nov 03 03:15:02.189429 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:15:02.189384 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:15:02.926587 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:02.926179 4992 generic.go:356] "Generic (PLEG): container finished" podID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerID="c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599" exitCode=60 Nov 03 03:15:02.926587 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:02.926402 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerDied","Data":"c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599"} Nov 03 03:15:02.926587 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:02.926468 4992 scope.go:122] "RemoveContainer" containerID="de32c20818cd80739bbd420d555bceb5689a545a09dead7aa42073784dca19bd" Nov 03 03:15:02.928573 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:02.927777 4992 scope.go:122] "RemoveContainer" containerID="c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599" Nov 03 03:15:02.928573 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:15:02.928086 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-q8grs_odh-ai-gateway-infra(b6b3ca6e-0f22-4292-837e-a3ad7858de2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" Nov 03 03:15:03.943318 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:03.943282 4992 scope.go:122] "RemoveContainer" containerID="c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599" Nov 03 03:15:03.943767 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:15:03.943522 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-q8grs_odh-ai-gateway-infra(b6b3ca6e-0f22-4292-837e-a3ad7858de2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" Nov 03 03:15:12.052869 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:12.052829 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb"] Nov 03 03:15:12.105000 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:12.104960 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fba94b6-krpjb"] Nov 03 03:15:12.593457 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:12.593422 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97d974a-4175-4db6-b53f-697e0eb685a9" path="/var/lib/kubelet/pods/e97d974a-4175-4db6-b53f-697e0eb685a9/volumes" Nov 03 03:15:13.587799 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:13.587761 4992 scope.go:122] "RemoveContainer" containerID="c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599" Nov 03 03:15:14.002446 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:14.002407 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerStarted","Data":"c74f883d64180ef633552265e46f8ac930e283c067d87e00e37fd6a1c354fe4a"} Nov 03 03:15:14.125260 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:14.125125 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" podStartSLOduration=14.125105098 podStartE2EDuration="14.125105098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:15:14.121791841 +0000 UTC m=+7049.700614428" watchObservedRunningTime="2026-11-03 03:15:14.125105098 +0000 UTC m=+7049.703927687" Nov 03 03:15:15.009243 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:15.009210 4992 generic.go:356] "Generic (PLEG): container finished" podID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerID="c74f883d64180ef633552265e46f8ac930e283c067d87e00e37fd6a1c354fe4a" exitCode=60 Nov 03 03:15:15.009688 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:15.009293 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerDied","Data":"c74f883d64180ef633552265e46f8ac930e283c067d87e00e37fd6a1c354fe4a"} Nov 03 03:15:15.009688 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:15.009360 4992 scope.go:122] "RemoveContainer" containerID="c14dfa794d59e49285a32fe8b01fe13e67ceed484156bee913566b98dc68f599" Nov 03 03:15:15.009799 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:15.009722 4992 scope.go:122] "RemoveContainer" containerID="c74f883d64180ef633552265e46f8ac930e283c067d87e00e37fd6a1c354fe4a" Nov 03 03:15:15.010010 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:15:15.009992 4992 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894595-q8grs_odh-ai-gateway-infra(b6b3ca6e-0f22-4292-837e-a3ad7858de2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" Nov 03 03:15:15.133551 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:15.133513 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:16.165001 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.164972 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:16.286540 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.286456 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\" (UniqueName: \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\") pod \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " Nov 03 03:15:16.286540 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.286505 4992 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\") pod \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\" (UID: \"b6b3ca6e-0f22-4292-837e-a3ad7858de2c\") " Nov 03 03:15:16.286885 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.286862 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca" pod "b6b3ca6e-0f22-4292-837e-a3ad7858de2c" (UID: "b6b3ca6e-0f22-4292-837e-a3ad7858de2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 03:15:16.289200 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.289163 4992 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz" pod "b6b3ca6e-0f22-4292-837e-a3ad7858de2c" (UID: "b6b3ca6e-0f22-4292-837e-a3ad7858de2c"). InnerVolumeSpecName "kube-api-access-fxbsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 03:15:16.388127 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.388089 4992 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbsz\" (UniqueName: \"kubernetes.io/projected/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-kube-api-access-fxbsz\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:16.388127 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:16.388122 4992 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b3ca6e-0f22-4292-837e-a3ad7858de2c-maas-api-service-ca\") on node \"ip-10-0-138-19.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 03:15:17.023339 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:17.023308 4992 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" Nov 03 03:15:17.023581 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:17.023307 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs" event={"ID":"b6b3ca6e-0f22-4292-837e-a3ad7858de2c","Type":"ContainerDied","Data":"c77de6e0035626f38d88ae1de9ccf8fd36e4f31628d6d73e351f0a612d609af7"} Nov 03 03:15:17.023581 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:17.023455 4992 scope.go:122] "RemoveContainer" containerID="c74f883d64180ef633552265e46f8ac930e283c067d87e00e37fd6a1c354fe4a" Nov 03 03:15:17.194884 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:17.194835 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:17.197390 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:17.197361 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894595-q8grs"] Nov 03 03:15:18.600415 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:18.600377 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" path="/var/lib/kubelet/pods/b6b3ca6e-0f22-4292-837e-a3ad7858de2c/volumes" Nov 03 03:15:32.152132 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:15:32.152091 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:15:38.453597 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:38.453550 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/2.log" Nov 03 03:15:39.096010 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:39.095978 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sjhd2_289f89f9-3d93-430e-9df8-0600cdd9b040/manager/0.log" Nov 03 03:15:39.986399 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:39.986365 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6nfb8_8b35e9d1-25de-4f51-af84-1b3f6a965f6c/server/0.log" Nov 03 03:15:40.221894 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:40.221851 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vpzrm_8972b343-b983-4b3c-879e-d4a6105ad81c/manager/0.log" Nov 03 03:15:41.225845 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:41.225814 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/1.log" Nov 03 03:15:42.368499 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:42.368459 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-050f0c64-t5nlb_1d0798d5-8bda-43a8-b8d4-b09f6a52ef0c/revoke-keys/0.log" Nov 03 03:15:42.581979 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:42.581945 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-132d83c3-mgvq2_fd23c856-24be-48c5-b79b-a704f7b6a8a8/revoke-keys/0.log" Nov 03 03:15:42.809177 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:42.809151 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-15816fcf-tlbmw_4355b819-0af1-4c14-ac17-972eb50913c7/revoke-keys/0.log" Nov 03 03:15:43.021129 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:43.021100 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-41cbdc9b-htm6g_8aa15afd-6dd1-47c2-8297-1b75085f1410/revoke-keys/0.log" Nov 03 03:15:43.252802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:43.252771 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-50e04e8c-qsv2m_433c37e6-30b5-41c6-a667-0b9eb50c96b9/revoke-keys/0.log" Nov 03 03:15:43.479962 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:43.479927 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-517d35eb-lmlzs_2ab9279c-6710-4cc0-9294-179dec61a3cf/revoke-keys/0.log" Nov 03 03:15:43.721501 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:43.721471 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-56d05f4f-2wbjj_4f4f4bb9-f716-41b7-a4c5-aae7bb9378c1/revoke-keys/0.log" Nov 03 03:15:43.954107 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:43.954080 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-673b9d5c-94jrt_bc4a41d6-fe27-4afb-9cc5-9bb1e40542fb/revoke-keys/0.log" Nov 03 03:15:44.189961 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:44.189927 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6776df4f-hrll2_50a948d7-e1bb-4f77-bb15-0dfe5cebf013/revoke-keys/0.log" Nov 03 03:15:44.424349 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:44.424316 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-73755308-rgl9c_1e3a064f-7938-43a3-9b0e-4b95b57fdcc8/revoke-keys/0.log" Nov 03 03:15:44.640926 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:44.640824 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7e786030-8pwvs_4521c884-90ce-420b-ac3f-37dd3cdae06d/revoke-keys/0.log" Nov 03 03:15:44.870884 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:44.870852 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91d8f222-bwg4k_168d9445-1e4f-4fd6-a05c-0c7ad890d519/revoke-keys/0.log" Nov 03 03:15:45.095099 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:45.095066 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-961a4af6-r4bnk_d365c88d-d2a7-4bd0-95ef-74f146bbb4b4/revoke-keys/0.log" Nov 03 03:15:45.322220 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:45.322181 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b28b10d8-28m2t_99c6b447-2479-4e83-a09a-4c5d8c114de3/revoke-keys/0.log" Nov 03 03:15:45.552212 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:45.552177 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b83bf462-925zk_0e3b4855-f92e-4431-b173-2fd7da55f166/revoke-keys/0.log" Nov 03 03:15:45.786335 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:45.786306 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fb200804-mvpzj_ef0ccdf2-4c4f-4711-bf50-9fce222c4c56/revoke-keys/0.log" Nov 03 03:15:46.522982 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:46.522938 4992 scope.go:122] "RemoveContainer" containerID="72c41e12cca739bc075eb38ec5836624e3b17a9927d6af905b95d55d7fd3f76c" Nov 03 03:15:46.532010 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:46.531986 4992 scope.go:122] "RemoveContainer" containerID="88512cc98fce61765a880640af064be4ef3759fa25fce3726417dbe69ff286d8" Nov 03 03:15:48.437295 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:48.437261 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nhqsx_22284cad-4990-4345-ae90-b13bdc75e40a/manager/0.log" Nov 03 03:15:48.995749 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:48.995717 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwl4r_9ffc6632-9a8c-46a2-8493-77016e28e030/kuadrant-console-plugin/0.log" Nov 03 03:15:50.072195 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:50.072139 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8f9fb_8b089aeb-b68c-4a95-b72e-daa34a2d0886/manager/0.log" Nov 03 03:15:51.249692 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:51.249643 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945svqcc_8bbb0964-a624-4157-83c3-f59eee954cbf/istio-proxy/0.log" Nov 03 03:15:52.458098 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:52.458066 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-4xmrn_5fb331db-b6cd-4bce-a099-f918904eec77/istio-proxy/0.log" Nov 03 03:15:53.549624 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:53.549594 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-nd2gf_33773d82-e52c-4603-899c-6935a2db93df/istio-proxy/0.log" Nov 03 03:15:53.883771 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:53.883668 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-nf7vp_0277aa76-953c-4912-861b-c1e841b0a47e/istio-proxy/0.log" Nov 03 03:15:54.947134 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:54.947105 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q_f843b8a6-9841-44ac-99ca-a1c4577301b5/istio-proxy/0.log" Nov 03 03:15:55.691724 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:55.691690 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85ff878dc9-x2zvj_835f0a42-f920-4d77-9178-4a174fe526c1/router/0.log" Nov 03 03:15:56.050942 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:56.050855 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs"] Nov 03 03:15:56.105950 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:56.105901 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-517d35eb-lmlzs"] Nov 03 03:15:56.593884 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:15:56.593847 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab9279c-6710-4cc0-9294-179dec61a3cf" path="/var/lib/kubelet/pods/2ab9279c-6710-4cc0-9294-179dec61a3cf/volumes" Nov 03 03:16:02.151426 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:16:02.151385 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:16:07.063349 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:07.063311 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c"] Nov 03 03:16:07.115097 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:07.115059 4992 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73755308-rgl9c"] Nov 03 03:16:08.593967 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:08.593930 4992 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3a064f-7938-43a3-9b0e-4b95b57fdcc8" path="/var/lib/kubelet/pods/1e3a064f-7938-43a3-9b0e-4b95b57fdcc8/volumes" Nov 03 03:16:09.477802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.477757 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kjgqj/must-gather-wmsxd"] Nov 03 03:16:09.478309 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478288 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478309 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478309 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478475 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478365 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478475 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478374 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478610 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478483 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478610 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478503 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478728 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478660 4992 cpu_manager.go:414] "containerMap: removing container" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478728 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478671 4992 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.478836 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.478767 4992 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b3ca6e-0f22-4292-837e-a3ad7858de2c" containerName="cleanup" Nov 03 03:16:09.483840 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.483813 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.538016 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.537975 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/must-gather-wmsxd"] Nov 03 03:16:09.586757 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.586720 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"openshift-service-ca.crt\"" Nov 03 03:16:09.586757 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.586748 4992 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kjgqj\"/\"default-dockercfg-9859t\"" Nov 03 03:16:09.586989 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.586808 4992 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"kube-root-ca.crt\"" Nov 03 03:16:09.615738 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.615682 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-must-gather-output\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.616152 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.615812 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kzm\" (UniqueName: \"kubernetes.io/projected/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-kube-api-access-x8kzm\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.717074 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.717030 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-must-gather-output\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.717266 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.717114 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kzm\" (UniqueName: \"kubernetes.io/projected/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-kube-api-access-x8kzm\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.717474 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.717450 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-must-gather-output\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.779829 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.779755 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kzm\" (UniqueName: \"kubernetes.io/projected/b65a2062-b4c3-4af0-9ddb-a617f059f3e9-kube-api-access-x8kzm\") pod \"must-gather-wmsxd\" (UID: \"b65a2062-b4c3-4af0-9ddb-a617f059f3e9\") " pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:09.798302 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:09.798268 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" Nov 03 03:16:10.170373 ip-10-0-138-19 kubenswrapper[4992]: W1103 03:16:10.170338 4992 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb65a2062_b4c3_4af0_9ddb_a617f059f3e9.slice/crio-0c73e284be190808875fad3aa9857432240945c5ffc12c7f1f69431e60daf7ac WatchSource:0}: Error finding container 0c73e284be190808875fad3aa9857432240945c5ffc12c7f1f69431e60daf7ac: Status 404 returned error can't find the container with id 0c73e284be190808875fad3aa9857432240945c5ffc12c7f1f69431e60daf7ac Nov 03 03:16:10.172626 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:10.172539 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/must-gather-wmsxd"] Nov 03 03:16:10.172823 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:10.172804 4992 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 03:16:10.327612 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:10.327550 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" event={"ID":"b65a2062-b4c3-4af0-9ddb-a617f059f3e9","Type":"ContainerStarted","Data":"0c73e284be190808875fad3aa9857432240945c5ffc12c7f1f69431e60daf7ac"} Nov 03 03:16:12.341715 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:12.341669 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" event={"ID":"b65a2062-b4c3-4af0-9ddb-a617f059f3e9","Type":"ContainerStarted","Data":"e1ce91a9d7c03ad1e998ca517672832f44919f680605f56f79d187c0df53c14e"} Nov 03 03:16:12.342252 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:12.341728 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" event={"ID":"b65a2062-b4c3-4af0-9ddb-a617f059f3e9","Type":"ContainerStarted","Data":"796f751a75411841b7d812c657a3d9e5bbf917c8338056da452ac4e8ac02d21c"} Nov 03 03:16:12.462747 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:12.462689 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kjgqj/must-gather-wmsxd" podStartSLOduration=2.062775147 podStartE2EDuration="3.462672657s" totalImagesPullingTime="1.39989751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:12.460441395 +0000 UTC m=+7108.039263982" watchObservedRunningTime="2026-11-03 03:16:12.462672657 +0000 UTC m=+7108.041495241" Nov 03 03:16:19.885301 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:19.885264 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xqklw_cbd9a86f-c50f-41b2-9c20-fe94fda357c1/global-pull-secret-syncer/0.log" Nov 03 03:16:20.518889 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:20.518853 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5kgrh_57f9e78d-c5d8-4af3-bbbb-15fddc658e70/connection-checker/0.log" Nov 03 03:16:20.981960 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:20.981930 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6pm5n_88af5e8b-13e7-47c8-9c9d-4c861f7c340b/connection-checker/0.log" Nov 03 03:16:21.429588 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:21.425205 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7rq8q_8e263b05-c71a-4d41-99d7-209ec4dbad6a/connection-checker/0.log" Nov 03 03:16:21.755018 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:21.754921 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/2.log" Nov 03 03:16:21.755188 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:21.755029 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/1.log" Nov 03 03:16:22.902479 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:22.902440 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sjhd2_289f89f9-3d93-430e-9df8-0600cdd9b040/manager/0.log" Nov 03 03:16:22.913931 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:22.913889 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tjn7g_add38c69-7c29-44f2-940f-061783060dcd/konnectivity-agent/0.log" Nov 03 03:16:23.228332 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:23.228295 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6nfb8_8b35e9d1-25de-4f51-af84-1b3f6a965f6c/server/0.log" Nov 03 03:16:23.730625 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:23.730581 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-19.us-west-2.compute.internal_b5ac7307a30e6ec216df9c122c45451f/haproxy/0.log" Nov 03 03:16:25.019321 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:25.019273 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vpzrm_8972b343-b983-4b3c-879e-d4a6105ad81c/manager/0.log" Nov 03 03:16:28.309822 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:28.309782 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/2.log" Nov 03 03:16:28.310451 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:28.309841 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/1.log" Nov 03 03:16:29.184365 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.184328 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sjhd2_289f89f9-3d93-430e-9df8-0600cdd9b040/manager/0.log" Nov 03 03:16:29.354374 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.354344 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nhqsx_22284cad-4990-4345-ae90-b13bdc75e40a/manager/0.log" Nov 03 03:16:29.433904 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.433865 4992 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8"] Nov 03 03:16:29.443454 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.443378 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.521513 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.521473 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8"] Nov 03 03:16:29.538991 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.538954 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5w9\" (UniqueName: \"kubernetes.io/projected/8cff7a62-b5b6-4c81-9b05-d34052107862-kube-api-access-qw5w9\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.538991 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.539003 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-lib-modules\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.539230 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.539178 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-podres\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.539230 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.539222 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-sys\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.539355 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.539334 4992 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-proc\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.563245 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.563213 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6nfb8_8b35e9d1-25de-4f51-af84-1b3f6a965f6c/server/0.log" Nov 03 03:16:29.640361 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640310 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-proc\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640592 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640425 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5w9\" (UniqueName: \"kubernetes.io/projected/8cff7a62-b5b6-4c81-9b05-d34052107862-kube-api-access-qw5w9\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640592 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640425 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-proc\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640592 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640485 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-lib-modules\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640592 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640586 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-lib-modules\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640851 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640677 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-podres\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640851 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640700 4992 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-sys\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640851 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640740 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-podres\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.640851 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.640835 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8cff7a62-b5b6-4c81-9b05-d34052107862-sys\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.704334 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.704254 4992 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5w9\" (UniqueName: \"kubernetes.io/projected/8cff7a62-b5b6-4c81-9b05-d34052107862-kube-api-access-qw5w9\") pod \"perf-node-gather-daemonset-gzmj8\" (UID: \"8cff7a62-b5b6-4c81-9b05-d34052107862\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:29.757360 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:29.757322 4992 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:30.147931 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.147893 4992 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8"] Nov 03 03:16:30.213112 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.213080 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwl4r_9ffc6632-9a8c-46a2-8493-77016e28e030/kuadrant-console-plugin/0.log" Nov 03 03:16:30.525891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.525249 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" event={"ID":"8cff7a62-b5b6-4c81-9b05-d34052107862","Type":"ContainerStarted","Data":"52fd714f3ce41f5908be2cac48f8315384fe78e3477bc0065e65845b457b142c"} Nov 03 03:16:30.525891 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.525401 4992 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" event={"ID":"8cff7a62-b5b6-4c81-9b05-d34052107862","Type":"ContainerStarted","Data":"2541ca5e3d8f5cb0103fdfd3bbe6ac9b0090b2711dbebc22536a8e63ecc71cda"} Nov 03 03:16:30.526497 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.525956 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:30.649370 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:30.649309 4992 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" podStartSLOduration=1.649294883 podStartE2EDuration="1.649294883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:30.645790274 +0000 UTC m=+7126.224612874" watchObservedRunningTime="2026-11-03 03:16:30.649294883 +0000 UTC m=+7126.228117470" Nov 03 03:16:31.089296 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:31.089260 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vpzrm_8972b343-b983-4b3c-879e-d4a6105ad81c/manager/0.log" Nov 03 03:16:31.613105 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:31.613067 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8f9fb_8b089aeb-b68c-4a95-b72e-daa34a2d0886/manager/0.log" Nov 03 03:16:32.156004 ip-10-0-138-19 kubenswrapper[4992]: E1103 03:16:32.155958 4992 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ktk4w" containerName="extractor" Nov 03 03:16:34.459370 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:34.459325 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/2.log" Nov 03 03:16:34.459802 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:34.459384 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-d6v2s_f592be42-f99a-48e7-adb0-314512784195/manager/1.log" Nov 03 03:16:35.352617 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:35.352073 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sjhd2_289f89f9-3d93-430e-9df8-0600cdd9b040/manager/0.log" Nov 03 03:16:35.823794 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:35.823762 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6nfb8_8b35e9d1-25de-4f51-af84-1b3f6a965f6c/server/0.log" Nov 03 03:16:36.546797 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:36.546760 4992 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-gzmj8" Nov 03 03:16:37.266428 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:37.266391 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vpzrm_8972b343-b983-4b3c-879e-d4a6105ad81c/manager/0.log" Nov 03 03:16:37.559395 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:37.559300 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-cs7fg_3aa0de45-e9ac-42fd-9ab0-a353fa05b2e4/cluster-monitoring-operator/0.log" Nov 03 03:16:41.534403 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:41.534364 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945svqcc_8bbb0964-a624-4157-83c3-f59eee954cbf/istio-proxy/0.log" Nov 03 03:16:42.720337 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:42.720299 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzg77_3930af2f-52ea-435f-8ae9-34678e6040c9/node-exporter/0.log" Nov 03 03:16:43.145391 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:43.145260 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzg77_3930af2f-52ea-435f-8ae9-34678e6040c9/kube-rbac-proxy/0.log" Nov 03 03:16:43.571534 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:43.571507 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nzg77_3930af2f-52ea-435f-8ae9-34678e6040c9/init-textfile/0.log" Nov 03 03:16:44.102148 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:44.102114 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-4xmrn_5fb331db-b6cd-4bce-a099-f918904eec77/istio-proxy/0.log" Nov 03 03:16:44.986232 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:44.986193 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-nd2gf_33773d82-e52c-4603-899c-6935a2db93df/istio-proxy/0.log" Nov 03 03:16:45.217239 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:45.217204 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-nf7vp_0277aa76-953c-4912-861b-c1e841b0a47e/istio-proxy/0.log" Nov 03 03:16:46.108787 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:46.108736 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cd79c685f-zf78q_f843b8a6-9841-44ac-99ca-a1c4577301b5/istio-proxy/0.log" Nov 03 03:16:46.561763 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:46.561731 4992 scope.go:122] "RemoveContainer" containerID="209da674543c88b6027207ec6e77efdbb792f1644ed2eed4129a8a02e07db8f6" Nov 03 03:16:46.781421 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:46.781386 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85ff878dc9-x2zvj_835f0a42-f920-4d77-9178-4a174fe526c1/router/0.log" Nov 03 03:16:48.232327 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:48.232291 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x548w_f1fc01e7-33b8-4129-a0f5-16ebe5c9786d/prometheus-operator/0.log" Nov 03 03:16:48.652232 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:48.652148 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-x548w_f1fc01e7-33b8-4129-a0f5-16ebe5c9786d/kube-rbac-proxy/0.log" Nov 03 03:16:49.079215 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:49.079162 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-zb5gh_faa8c85c-155f-4121-970b-9ea01d3188b9/prometheus-operator-admission-webhook/0.log" Nov 03 03:16:52.372675 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:52.372636 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nhqsx_22284cad-4990-4345-ae90-b13bdc75e40a/manager/0.log" Nov 03 03:16:52.788167 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:52.788136 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pwl4r_9ffc6632-9a8c-46a2-8493-77016e28e030/kuadrant-console-plugin/0.log" Nov 03 03:16:52.978408 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:52.978373 4992 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945svqcc_8bbb0964-a624-4157-83c3-f59eee954cbf/istio-proxy/0.log" Nov 03 03:16:53.066527 ip-10-0-138-19 kubenswrapper[4992]: I1103 03:16:53.066446 4992 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50e04e8c-qsv2m"]